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Note: Throughout this report, references to “project schools,” “MLD Project schools,” or 
“intervention schools” mean the schools that received all project services and activities. In 


contrast, schools that participated in data collection, but did not receive any project services or 


activities, are referenced only as “comparison schools.” 


EXECUTIVE SUMMARY 


Executive summary 


MLD PROJECT OVERVIEW 

The Middle-Grades Leadership Development (MLD) Project was designed to develop 
principal leaders and leadership teams who create high-performing middle-grades schools. 
Developed by the National Forum to Accelerate Middle-Grades Reform, the four-year project was 
funded in 2013 by a U.S. Department of Education Investing in Innovation (i3) development grant. 
The project was implemented in Kentucky by the Kentucky Middle School Association (KMSA) and 
in Michigan by the Institute for Excellence in Education (IEE). The Center for Prevention Research 
and Development (CPRD) at the School of Social Work, University of Illinois served as the 
evaluator. 


The MLD Project used an extensive set of school improvement supports, including: 


creating a vision using the Schools to Watch ; : 
or bee “The grant allowed for the implementation of 
(STW) criteria; engaging in an assessment and 

. . distributive leadership. Through this, we were 
planning process for improvement; STW ate 
; eee able to develop a common vision throughout 
leadership coach; principal mentor; STW mentor i ; 
: ‘ our faculty about what good instruction looks 
schools; leadership team; networking ; : 
es ; like, sounds like, and how we want our 
opportunities; and focused professional ; 
: ; : classrooms to function. We changed our 
development. Project planning began in January 


professional development model to one where 


2014 with the national and state partners, and 
the 12 MLD Project middle-grades schools (6 in 
Kentucky and 6 in Michigan) began receiving 
services in July/August 2014 through May 2017. 


we learn from each other. This allowed a team 
of leaders to be developed around the school. 
We truly have become a faculty of leaders.” 

- MLD Principal 


EVALUATION DESIGN 

The evaluation of the MLD Project used a quasi-experimental design (QED) with matched 
comparison schools. Twelve middle-grades schools in Kentucky and Michigan (6 schools per state) 
received the project treatment. Each of the twelve treatment schools were matched with two to 
four comparison schools from the same state and with a similar profile of achievement test scores, 
No Child Left Behind (NCLB) Adequate Yearly Progress (AYP) status and history, and school and 
student demographics. A cohort of 6 graders from the 50 schools (12 treatment and 38 
comparison) were followed for three years, and math and ELA/reading state achievement test 
scores (Kentucky Performance Rating for Educational Progress and Michigan Student Test of 
Educational progress) were the primary outcomes, along with principal effectiveness scores via 
Vanderbilt’s Assessment of Leadership in Education (VAL-ED). Intermediate outcomes were tracked 
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with mixed methodologies consisting of data collected via the National Forum’s STW Rating 


Rubric, CPRD’s School Improvement Self-Study Teacher Survey, focus groups with coaches and 
principals, a principal survey, and CPRD’s Coach’s Log. The project’s impact — defined as the 
difference between the project and comparison schools on achievement test scores - was 
estimated using a hierarchical linear model (HLM) approach. The overall intervention effect 
between the project and comparison schools on achievement test scores was examined. 


MLD PROJECT PRINCIPALS 

The 12 MLD Project principals were experienced educators with an average of 17.6 years 
of work in the field of education and an average of 13.2 years working in the middle grades. 
Principals reported an average of 8.5 years serving as a principal with 5.1 of those years as a 
principal in their MLD Project school. All MLD Project school principals had a school 
administrator/principal certification. A total of 8 principals had a Master’s degree in Education 
and the remaining 4 had an Educational Specialist degree. Although very few received pre-service 
specialized training in the middle-grades education, all participated in professional development 
on middle-grades instruction and organizational needs during their years as a teacher or 
administrator. There was collective agreement that the MLD Project was helping to improve 
leadership at their school, both for themselves as well as for their faculty 


IMPLEMENTATION FINDINGS 

The eight key programmatic components (create a powerful vision using STW; engage in 
assessment and planning to identify needs, develop goals, and implement an action plan; STW 
leadership coach; principal mentor; STW mentor school; leadership team; networking 
opportunities; and focused professional development) of the MLD Project were implemented with 
fidelity at the majority of project schools. There was some variability in implementation by school, 
with four schools having slightly higher and more consistent overall implementation. In focus 
group interviews with coaches, they commented that several schools had a higher level of 
readiness for making improvements at the start of the project and their districts were highly 
involved and supportive of their goals. 


INTERMEDIATE OUTCOME FINDINGS 

Evidence indicates that MLD Project schools significantly improved their implementation 
of the STW Rubric criteria for high performing middle-grades schools, collaboration practices, 
teacher efficacy, and middle-grades classroom instructional practices. 
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e STW Rubric - Higher implementation of strategies to support academic excellence, 
developmental responsiveness, social equity, and organizational structures. 


e Collaboration practices - Average Staff Ratings on STW Rubric 
5 4 
Increased team practices, 
quality of collaborative 
3 
interactions, and team decision 
making. : 
e Teacher efficacy and 
: ee 1 
collective responsibility - 
Improvements in teachers’ : 
individual and collective Academic Excellence Developmental Social Equity Organizational 
. Responsiveness Structures 
commitment to the success of 142014/15 2015/16 2016/17 
their students. 
4=High quality; 3=Good quality; 2=Fair quality; 1=Poor quality ps.001 


e Middle-grades instructional practices - Increased recommended practices such as 
small group instruction, integration and interdisciplinary practices, and critical thinking 
practices. 


FINAL OUTCOME FINDINGS 
MLD Project principals improved their leadership skills and behaviors throughout the 
project, with principals demonstrating the greatest growth in the areas of culture of learning and 
professional behavior, quality instruction, performance accountability, and high standards for 
student learning. Overall principal 


effectiveness, as measured by all “The overall impact of the grant has been very 
respondent groups (i.e., principal, positive. It has given us purpose, direction, and 
teachers, supervisor), improved from two guidance to improve what we do for the overall 
principals rated as proficient or growth of the whole student.” 


distinguished in Year 1 of the project to - MLD Principal 
nine principals by the end of Year 3. Based 

on the percentile score of the principal’s mean score in comparison to a national sample of 
principal means, all twelve principals grew as leaders when compared to the national sample. 


A comparison of the MLD project schools to the state at the 8 grade level in 2017 (after 
the three-year project implementation) showed that seven schools had larger reading growth than 
the state and three schools had larger math growth than the state. Although these are positive 
trends in achievement, further analyses were conducted on the cohort of students tracked for 
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three years at MLD Project schools as compared to a cohort at comparison schools using 
Hierarchical Liner Models (HLMs). The series of 2-level models, where students were nested 
within schools, were executed to analyze the intervention effect on student achievement. The final 
models suggest no significant intervention effect on either ELA/reading scores or math scores. In 
other words, the students that received the MLD Project intervention performed the same on both 
the ELA/reading test and the math test as the comparison students after three years. Neither the 
p-values nor effect sizes suggest a significant intervention effect on achievement. Even though the 
impact analyses did not find an overall intervention effect on achievement, it should not be 
interpreted to mean that the MLD Project intervention was not effective, but that project schools 
may need additional time to fully implement and refine the model with middle-grades students to 
advance achievement scores. We can further hypothesize that the positive changes on school 
organization and climate factors such as the STW criteria, collaboration, teacher efficacy, middle- 
grades instructional practices, and leadership are a positive precursor to improvements in 
achievement since achievement does not improve without corresponding improvements in the 
teaching and learning environment. 


LESSONS LEARNED 

There are a myriad of lessons learned from the MLD Project about how middle-grades 
schools embark on leadership improvements and the implementation of a collaborative 
leadership model, including: 


e The STW criteria provided a guiding vision to MLD Project schools for what a high performing 
middle-grades school should be. It served as a framework, common language, and pathway to 
improvement. 


e The adoption and use of a continuous school improvement model where data was used at 
every stage of a cycle to inform planning, set targeted goals, reflect on and evaluate progress, 
and refine implementation, was a key aspect remain focused on improvement. 


e The implementation ofa collaborative leadership structure empowered teachers to take on 
shared leadership roles and build capacity. 


e The development of strong leadership teams that were engaged in collaboration and focused 
on improvement was a key to facilitating change. Leadership teams examined data to drive 
actions, had a continuous improvement approach, communicated with the whole faculty, built 
capacity for sustainability, and evolved as they learned from challenges. 
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e Middle-grades principals need support and resources to assist them with transforming their 


leadership practices, beliefs, and behaviors. rom ; ; ; 
| think for me the biggest piece was just 


e Visits to STW schools made a powerful impact to increase my knowledge about what 
works in the middle grades and how we 
can best implement those practices in 


; my school setting.” 
another school, and share a common experience. - MLD Principal 


because they allowed teachers to observe best 
practices, gain knowledge about the successes at 


e Participation in the Forum’s STW network at the state and national levels provided schools 
with knowledge, resources, and supportive professional development opportunities. 


e It was important to build a positive, supportive, reflective, and student-centered school 
culture. The contextual changes in culture were viewed as pre-requisites for leadership 
change, collaboration opportunities, and instructional improvements. 


e Cultivating school district 
“We’re moving from an adult-centered to a 


student-centered facility and programs. The piece 
that has really bought us growth from a teaching 


involvement and support of the 
school’s improvement plans from the 


beginning and maintaining it standpoint is active engagement strategies for 
throughout the project was an youngsters in the classroom and that is really 
important part of successful elevating the teaching and learning. That is critical. 
improvements. They absolutely engage kids. We’ve got a group of 
kids that have to be active, so might as well channel 
e Participation in mock STW visits was it in a useful way.” 
the activity that taught schools the - MLD Leadership Coach 


most about their improvement 

progress and goals. Mock STW visits involved the MLD Project schools hosting a national team 
of STW-trained visitors whose purpose was to spend a day touring the school, meeting with 
administrators and teachers, observing classrooms, and interviewing students and parents, 
then providing feedback to the school leadership on future direction. 


CONCLUSION 

The results of the evaluation of the MLD Project are positive in that they highlight 
numerous gains and improvements at MLD Project schools. The results also provide unique 
insight into a middle-grades school improvement project focused on principal leaders and 
collaborative leadership. The figure below depicts the key supports, activities, and practices 
implemented at MLD Project schools that were the most impactful on building middle-grades 
leadership effectiveness. They provide a road map for other middle-grades schools and principals 
that are struggling to improve. With key supports, activities, and practices at the school level, by 


Page 5 MLD Project Final Evaluation Report CPRD, 2018 


EXECUTIVE SUMMARY 


the principal, through a collaborative leadership structure and in a positive teaching and learning 
culture, middle-grades leadership is more effective. 


School Environment: Principals: 
- Adoption of STW guiding vision for middle- - Knowledgeable about young adolescents 
grades high performance - Committed to developmentally appropriate 
- Use of STW criteria for continuous middle-grades practices and structures 
improvement planning and monitoring - Serve as an instructional leader 
Intentional use of examining school data to - Connected to a network for resources, best 
improve teaching practices practice, and visits to other schools 


- Use of reflective practices 


Middle-Grades 


Leadership 


Collaborative Leadership: Teaching and Learning Culture: 
- Development of teacher leaders - Student-centered learning for all 
- Implementation of structures for collaboration - Consistent and high expectations 
- Establish time for regular collaboration - Build strong relationships with students 
- Build shared capacity - Engaging and rigorous instruction 


- Involve district in improvement efforts - Growth mindsets among teachers and 
students 
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MLD Project 


The Middle-Grades Leadership Development (MLD) Project was designed to develop 
principal leaders and leadership teams who create high-performing middle-grades schools. 
Developed by the National Forum to Accelerate Middle-Grades Reform, the four-year project was 
funded in 2013 by a U.S. Department of Education Investing in Innovation (i3) development grant. 
The project was implemented in Kentucky by the Kentucky Middle School Association (KMSA) and 
in Michigan by the Institute for Excellence in Education (IEE). The Center for Prevention Research 
and Development (CPRD) at the School of Social Work, University of Illinois served as the 


luati tner for the MLD Project. 
Soe yee ae en “It’s really easy for everybody to focus 


The MLD Project sought to improve on academic achievement. Everything is 
principal effectiveness in order to improve the driven by test scores. J think teachers 
academic achievement of high-needs, middle- have really appreciated having the four 
grades students at 12 schools, through the areas spelled out in the STW Rubric 
development of principal and leadership team criteria, knowing that in middle school 
skills and behaviors to create high-performing, it is so important to be in tune with 
middle-grades schools and through improved student development.” 
climate and culture for learning. - MLD Principal 


The intervention intended to accomplish this goal using the framework of the National 
Forum to Accelerate Middle-Grades Forum’s Schools to Watch (STW) vision and criteria with the 
following program objectives: 


1) Improve principal and leadership team skills, behaviors, and practices for continuous 
improvement; 

2) Improve school implementation of the four key criteria of the STW vision (academic 
excellence, developmental responsiveness, social equity, and organizational structures 
and processes) by implementing programs, practices, and organizational structures 
consistent with the vision; 

3) Improve school climate and culture for learning by engaging in the STW network to 
exchange ideas, solve problems, and discuss practice; and 

4) Improve student achievement outcomes. 


Project planning and development began in January 2014 with the national and state 
partners. The 12 MLD Project middle-grades schools (6 schools in Kentucky and 6 schools in 
Michigan) began receiving services in July/August 2014. 
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INTERVENTION ACTIVITIES/KEY PROGRAM PIECES 

The MLD Project used an extensive set of school improvement supports, including a 
Schools to Watch (STW) leadership coach, principal mentor, STW mentor schools, focused 
professional development, and networking opportunities. Through these supports and activities, 
schools created a powerful vision for high performance using the STW criteria; engaged in an 
assessment and planning process to develop and implement a formal action plan; and used 
intervention supports and services to develop leadership capacity and sustainability in 
professional learning communities. The project was guided by a logic model (Figure 1). 


Figure 1. MLD Project Logic Model 


National Forum’s Middle-Grades Leadership Development (MLD) 


Logic Model 
Program Intermediate Long Term 
Interventions Outcomes Outcomes 


*Schools create a powerful vision +Increased confidence +Improvement in principal 
statement for high performance using an Orinci Ae effectiveness scores. 
the Schools to Watch (STW) criteria. rabies a 


+Schools engage in assessment and 
planning to identify needs, develop 
goals, and develop and implement a 
formal action plan. 


+*MLD leadership coach works with +Improvements in *Improvement in student 
leadership team. professional collaboration achievement test 
experiences. scores. 


+Principal mentor works with principal. 


+Network mentor school works with 
school. 


P 5 + Improvements in 
*School engages in leadership professional work climate 
professional learning community. and teaching efficacy. 


+ School participates in networking 
experiences. 


+School receives regular and systematic 
professional development trainings 
based upon needs and data from the 
evaluation process. 


EXPECTED OUTCOMES 
It was hypothesized that schools engaged in the MLD Project would experience 
improvements in the following intermediate outcomes: increased confidence among faculty in the 
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principal’s leadership skills; improvements in professional collaboration experiences; and 
improvements in professional work climate and teaching efficacy. Ultimately, success would be 
measured on the long-term outcomes of the project: improvements in principal effectiveness 
scores and improvements in student achievement test scores. 


Principal effectiveness was measured with the VAL-ED Leadership Assessment. Student 
achievement was measured with state standardized achievement tests in math and ELA/reading 
(K-PREP test in Kentucky, M-STEP test in Michigan). 


PROJECT SCHOOLS 

The twelve schools that participated in the MLD Project were located in primarily rural or 
small town areas of Kentucky and Michigan, each serving an average of 424 students (Table 1). 
The average percentage of students receiving free or reduced-priced lunch across the 12 schools 
was 63%. Approximately 18% of students were minorities and only 2% were English language 
learners. 


Table 1. MLD Project School Demographics 


Percent Percent 
res ; Grade | Enroll- Free/ Percent English 
Distuicts pie otate Levels | ment Reduced | Minority | Language 
Lunch Learners 
Estill County Middle | Estill County Schools | Irvine, KY 542 72% 1% 
School Rural 
Garrard Middle Garrard County Lancaster, 545 63% 10% 
School Schools KY (Rural) 


Georgetown Middle Scott County Schools | Georgetown, 485 57% 21% 
School KY 
(Suburban) 


Mayfield Middle Mayfield Mayfield, KY 338 80% 18% 
School Independent Schools | (Small 

Rowan County Rowan County Morehead, 737 62% 7% 
Middle School Schools KY (Small 

Union County Middle | Union County Morganfield, 59% 15% 
School Schools KY (Rural) 

Kentucky Totals 3, Rea | 64% | 12% | 


Grant Middle School | Grant Public School Grant, MI - 55% 25% 
District Rural 

Marshall Greene Birch Run Area Birch Run, 5-8 408 50% 9% 

Middle School Schools MI (Rural) 
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Mt. Morris Junior Mt. Morris Mt. Morris, 430 75% 23% 
High School Consolidated Schools | MI 

(Suburban/ 

Urban) 


Rural 

(Rural) 
Ea El 
School Academy School Academy (Urban) 
fia [Michigan Totals ssid | 1941 | 941 | 63% ay) 


Notes: Data from 2014/15 (Year 1). Enrollment includes only grades served (6-8). 
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Evaluation Design 


The Center for Prevention Research and Development (CPRD) at the School of Social Work, 
University of Illinois served as the evaluation partner for the MLD Project. The evaluation used a 
quasi-experimental design (QED) with matched comparison schools. Twelve middle-grades 
schools in Kentucky and Michigan (6 schools per state) received the project treatment. Each of the 
twelve treatment schools were matched with two to four comparison schools from the same state 
and with a similar profile of achievement test scores, No Child Left Behind (NCLB) Adequate 
Yearly Progress (AYP) status and history, school enrollment size, and school level demographics. A 
cohort of 6 graders from the 50 schools (12 treatment and 38 comparison) were followed for 
three years, and math and ELA/reading state achievement test scores were the primary outcomes. 
Intermediate outcomes were also tracked at intervention schools with mixed methodologies 
consisting of data collected via Vanderbilt’s Assessment of Leadership in Education (VAL-ED), the 
National Forum’s STW Rating Rubric, CPRD’s School Improvement Self-Study Teacher Survey, focus 
groups with coaches and principals, a principal survey, and CPRD’s Coach’s Log. The data analysis 
plan for assessing the impact of the project used a hierarchical linear model (HLM) approach. The 
overall intervention effect between the project and comparison schools on achievement test 
scores was examined. 


SAMPLE 

Fifty public middle-grades schools serving grades 6t to 8 in Kentucky and Michigan (6 
treatment and 19 comparison schools per state) participated in the evaluation. All schools served 
high-need student populations with suppressed achievement test scores. A cohort of 6 graders 
were followed at treatment schools for three years, comparing their outcomes to a cohort at 
comparison schools. A total of 1,304 treatment and 4,433 comparison students participated in the 
cohort. Comparison schools were “business as usual” middle-grades schools that did not receive 
the MLD Project intervention, i.e., levels of support, training, or mentoring for their principals and 
school leadership. 


Within Kentucky and Michigan, a pool of potentially participating middle-grades schools 
was developed. The treatment sample was selected using detailed criteria that each school must 
meet in order to be considered for inclusion in the project, including: 1) Public school; 2) Middle- 
grades school with a grade configuration of at least 6 through 8"; 3) Serve high-need student 
populations (e.g., students at risk of educational failure, such as students who are living in poverty, 
who are English language learners, who are performing below grade level); and 4) Suppressed 
achievement scores. This criteria was used to identify a pool of eligible schools and recruit their 
participation. It is considered a convenience sample with well-defined criteria. 
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The comparison schools were selected from the same pool of eligible schools used to 
select the treatment schools. The matching was conducted using propensity scores at the school 
level within each state from the pool of eligible schools. Schools were matched on the following 
variables: state, enrollment, student demographics, achievement test scores, and NCLB AYP status 
and history. When possible, matches were explored within the same district as treatment schools. 
Schools were matched first on school level achievement test scores and second on NCLB AYP 
status and history. Once achievement test scores and NCLY AYP status and history were matched, 
schools were matched on the remaining variables in the list above. For some schools, matches 
were difficult on the remaining list of multiple variables, thus we allowed a 5% to 10% margin of 
difference as the target. Baseline equivalence tests between the treatment and comparison schools 
were conducted based on their 5 grade achievement scores (i.e., the year before the intervention 
began) and other available demographic variables on the final analytic sample to verify the 
equivalence between treatment and comparison schools (see pages 36-37 for more information). 
Treatment schools appeared similar compared to the comparison schools based on their 5 grade 
achievement scores as well as the other student level demographic variables. Yet to reduce the 
student level variances, the baseline scores and other covariates were used to adjust the 
evaluation models. 


All students enrolled in treatment schools and comparison schools as 6% graders in 
2014/15 were part of the cohort of students that was followed for three years. Students who 
entered the schools later in 2014/15 or in between project years were excluded. Students with no 
pretest score (5th grade) were also excluded. A total of 1,304 6 graders from twelve treatment 
schools and 4,433 6t graders from the thirty-eight comparison schools over a three year period 
were tracked. 


RESEARCH QUESTIONS 

The purpose of the impact study was to determine if the MLD Project intervention 
significantly improved student math and ELA/reading achievement after three years of exposure, 
relative to comparison students. There are two confirmatory research questions and one 
exploratory research question for the impact study. 


Impact Study Confirmatory Research Questions: 

1) Did the MLD Project intervention have an effect on the math achievement of 8th grade 
students, after three years of intervention, as compared to business-as-usual 
condition? 


2) Did the MLD Project intervention have an effect on the ELA/reading achievement of 8% 
grade students, after three years of intervention, as compared to business-as-usual 
condition? 
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Impact Study Exploratory Research Question: 


3) Did the MLD Project intervention improve principal leadership practices, 
organizational capacity, and school influences? 


The purpose of the implementation study was to document the fidelity of implementation 
of the MLD Project at each treatment school, assessing whether the intervention was implemented 
as intended. There is one exploratory research question for the implementation study. 


Implementation Study Exploratory Research Question: 


1) Was the MLD Project intervention implemented with fidelity at each treatment 
school? 


DATA SOURCES 

Data sources for the evaluation included mixed methodologies. Math and ELA/reading 
achievement test scores were collected for a cohort of 6" graders from the 50 schools (12 
treatment and 38 comparison) who were followed for three years (Table 2). Intermediate 
outcomes were tracked at treatment schools with data collected via Vanderbilt’s Assessment of 
Leadership in Education (VAL-ED), the National Forum’s STW Rating Rubric, CPRD’s School 
Improvement Self-Study Teacher Survey, focus groups with coaches and principals, principal survey, 
and CPRD’s Coach’s Log. 


Math and ELA/reading achievement test scores were collected from each state. For project 


and comparison schools in Kentucky, the Kentucky Performance Rating for Educational Progress 
(K-PREP) achievement test was used. For project and comparison schools in Michigan, the 
Michigan Student Test of Educational Progress (M-STEP) test was used. Both represent the state 
approved standardized achievement test. The achievement data was obtained from the state 
departments of education via a data sharing agreement during the spring of each project year, 
beginning with the baseline 5‘ grade test scores in 2014. 


VAL-ED (Vanderbilt Assessment of Leadership in Education) is a research-based 


evaluation tool created in 2006 by Vanderbilt University (with support from the Wallace 
Foundation and the U.S.D.E. Institute for Education Sciences) that measures the effectiveness of 
school principals using 360 degrees of feedback from teachers, supervisors, and principals. The 
VAL-ED assesses principals in the following six core components of leadership and six key 
processes of leadership: 
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e Core Components of Leadership: high standards for student learning, rigorous 
curriculum, quality instruction, culture of learning and professional behavior, connections 
to external communities, and performance accountability; and 


e Key Processes of Leadership: planning, implementing, supporting, advocating, 
communicating, and monitoring. 


The valid and reliable results of the VAL-ED Assessment allow the principal to compare 
their ratings on each key component or process against the ratings given by teachers and the 
supervisor. In this way, the principal receives informative feedback about their leadership 
behaviors in order to develop a professional growth plan. Additionally, the report contained norm- 
referenced and criterion-referenced scores for assessing leadership. VAL-ED was administered at 
project schools annually. 


STW Rating Rubric is a tool developed by the National Forum in 1998 that measures the 
implementation of the criteria associated with the four components of the STW Program. The 
rubric is used by middle grades schools to study and rate their practices as part of a continuing 
improvement process as well as part of a mandatory self-rating for schools interested in applying 
for a STW designation. The four STW program components measured by the rubric include: 


1. Academic Excellence - Schools challenge students to use their minds well; 


2. Developmental Responsiveness - Schools are sensitive to the unique 
developmental challenges of early adolescence; 


3. Social Equity - Schools are democratic and fair, providing every student with high- 
quality teachers, resources, learning opportunities and supports; and 


4. Organizational Structures and Processes - Schools establish norms, structures, and 


organizational arrangements to support and sustain their trajectory toward 
excellence. 


For each of the four STW program components, teachers at treatment schools rated their 
school’s level of implementation by responding to 37 general criteria and 100 concrete examples 
of excellence. Teachers used a metric ranging from one to four where: 4 = High quality, complete, 
mature, and coherent implementation - NEARLY PERFECT, LITTLE ROOM FOR IMPROVEMENT; 3 
= Good quality, maturing but not fully implemented by all - GOOD QUALITY BUT STILL ROOM FOR 
REFINEMENT AND IMPROVEMENT; 2 = Fair quality, mixed implementation, immature practice, 
sporadic by some - SIGNIFICANT IMPROVEMENT NEEDED; and 1 = Poor quality, low level of 
implementation, new program - CONSIDERABLE PLANNING, CONSENSUS BUILDING AND 
IMPROVEMENT NEEDED. The STW Rating Rubric was administrated at project schools annually. 


Page 14 MLD Project Final Evaluation Report CPRD, 2018 


EVALUATION DESIGN 


School Improvement Self-Study Teacher Survey is part of a data collection system of 
surveys for middle-grades schools, developed by CPRD in 1990. The surveys are grounded in 


research and have been used with more than a thousand schools, with results widely disseminated 
(Flowers & Mertens, 2003; Flowers, Mertens, & Mulhall, 1999, 2000a, 2000b, 2002, 2003, 2007; 
Mertens & Flowers, 2003, 2006; Mertens, Flowers, Hesson-MclInnis, & Bishop, 2006, 2007). The 
teacher survey is comprised of numerous constructs related to the teaching and learning process 
including: interdisciplinary teaming practices, quality of team interactions, team decision making, 
work climate, collective teacher efficacy, teacher decision making, administrative leadership, and 
classroom instructional practices. The items that make up each construct on the teacher survey 
were combined and scale scores were calculated based on Cronbach’s alpha (.76 to .96). The Self- 
Study Teacher Survey was administrated at project schools annually. 


Focus Groups were conducted with a sample of project principals and leadership coaches 
annually in order to ensure that the voice of each principal and coach was represented over the 
course of the MLD Project. The purpose of the focus groups was to provide formative results 
regarding the experiences of implementation and to assess the multi-layered system of support 
(e.g., coaching, mentoring activities, networking, etc.). Focus group topics changed each year to 
address the different stages of implementation and to answer questions not addressed by other 
data sources. Key areas discussed included leadership development (i.e., principal, leadership 
team, teacher leaders), barriers to implementation and lessons learned, assessing which program 
interventions had the greatest impact, building capacity for sustainability, and the pathway to STW 
School designation. 


A Principal Survey was conducted with all treatment school principals during the final 
year of the grant. The purpose of the survey was to collect information about each principal’s 
experiences in education and the middle grades, the helpfulness of the grant services they 
received, and the impact the grant had on their school. All twelve principals completed the survey. 
These data were used in analyses in order to disaggregate data by principal experience and 
certification in order to understand variances in outcomes across the treatment schools who 
participated in the MLD Project. 


Coach’s Log is an electronic data collection system for leadership coaches to complete after 
each visit or activity with a project school. The purpose of the log was to document the number of 
visits made to schools, the purpose of the visit (e.g., training, professional development, reviewing 
data, etc.), the outcome of the visit (e.g., goals set, action plan distributed, etc.), as well as to log the 
improvement progress of the school. The coach’s log was completed by leadership coaches on an 
ongoing basis throughout the MLD Project. 
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Table 2. MLD Project Data Components and Sample Sizes 


Data Component School Year 1 School Year 2 School Year 3 
(2014/15) (2015/16) (2016/17) 

Achievement Test 1,304 project students & 4,433 comparison students tracked for 3 years 
VAL-ED 300 assessments 292 assessments 279 assessments 
STW Rubric 355 staff 347 staff 322 staff 
Teacher Survey 323 teachers 270 teachers 288 teachers 
Coach’s Log 191 Total Contacts 268 Total Contacts 256 Total Contacts 
Focus Groups 7 Principals; 6 Coaches | 6 Principals; 5 Coaches | 6 Principals; 6 Coaches 


Feedback reports were disseminated annually for all data sources to the various project 
stakeholders and to treatment schools. Aggregated achievement data was reported for groups of 
students (i.e., grade, school, and cohort-level data) to state leaders and project leadership team to 
monitor the impact of the intervention. Reports from all four of the process measurement tools 
(i.e., STW Rubric, Self-Study Teacher Survey, Coach’s Log, and Focus Groups) were disseminated 
annually to project schools, leadership coaches, state leaders, and the project leadership team for 
use in monitoring improvement progress, setting goals, and refining the intervention. School, 
state, and project-level feedback reports were provided for the STW Rubric and Self-Study Teacher 
Survey. State-level reports were provided for the Coach’s Log. Focus group reports were 
aggregated across all participants. Longitudinal data was provided for every year of participation 
in the STW Rubric, Self-Study Survey, and Coach’s Log. 


DATA ANALSIS APPROACHES 

The 2014/15 school year was the first year that the MLD Project was implemented in 
treatment schools, and the treatment extended through spring 2017. Treatment and comparison 
students in sixth grade in 2014/15 were followed (Table 3) through spring 2017 (the end of their 
8th grade year). Math and ELA/reading achievement state standardized test scores were collected 
in spring 2015, 2016, and 2017. The 2017 scores served as the outcomes for the confirmatory 
research questions. 


Table 3. MLD Project Student Cohort Tracked 


School Year 1 School Year 2 School Year 3 
(2014/15) (2015/16) (2016/17) 


Cohort Tracking 
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For the impact study, hierarchical linear models (HLMs), clustering students within 
schools, were used to analyze the intervention effect on student achievement scores. The 
dependent variables were related to two outcome domains: 8th grade achievement test scores in 
mathematics and 8 grade achievement test scores in ELA/reading, after following the cohort of 
6th grade who continued to belong to their respective treatment schools and hence receive three 
years of the intervention. For schools in Kentucky, the Kentucky Performance Rating for 
Educational Progress (K-PREP) achievement test was used. For schools in Michigan, the Michigan 
Student Test of Educational Progress (M-STEP) test was used. Both are standardized tests and 
hence there are no concerns regarding reliability, over-alignment with the intervention, or 
inconsistent data collection. However, to allow comparison of scores across different state tests 
and over years, each state’s achievement scores were converted into z-scores (standardized to the 
mean and SD of the state test for each year for that subject) using the test technical manual. The 


x-U : ; ; ; 
z-score formula used was: Z = i. where x is the student’s scale score, is the corresponding state 


mean, and SD is the corresponding state standard deviation matched for the corresponding year. 


To evaluate the impact of the intervention, an independent variable was generated that 
identifies the treatment and comparison schools coded as 1 and 0, respectively. The following 
covariates were included as part of adjusting for baseline differences with the goal of improving 
the accuracy of the coefficients and to be aligned with the covariates used in propensity scores 
matching while selecting the comparison schools. 


Student-level covariates — 5t* grade pre-test achievement scores, race/ethnicity (white vs. 
not-white, African American vs. not African American, Hispanic vs. non-Hispanic), gender, LEP 
status, free/reduced lunch status, and special education status; and 


School-level covariates - free/reduced lunch, enrollment, NCLB AYP status, and school 
demographics. 


To estimate the mean baseline difference, the same modeling approach was used for 
estimating differences on outcomes, but without any covariates. For the implementation study, 
each key project component was scored using multiple indicators. An implementation score was 
determined for each indicator. All indicators except one were measured at the school level. The 
one indicator not measured at the school level was measured at the teacher level instead. Once the 
indicator level scores were calculated for each project component, the individual indicator scores 
were scored at the key project component level to arrive at a component level implementation 
score. Finally, for each component, a threshold was established for assessing fidelity of 
implementation across the sample (e.g., 85% percent of schools must have high implementation) 
in order to determine if fidelity of each component was met. 
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MLD Project Principals 


In the final year of the grant, the evaluation team requested that each MLD Project 
principal complete a short survey. The survey asked about their experiences in education and the 
middle grades, the helpfulness of the grant services they received, and the impact the grant had on 
their school. 


PRINCIPAL PROFESSIONAL EXPERIENCES 

The MLD Project principals were experienced educators with an average of 17.6 years of 
work in the field of education and an average of 13.2 years working in the middle grades (Table 4). 
Principals reported an average of 8.5 years serving as a principal with 5.1 of those years as a 
principal in their MLD Project school. 


Table 4. MLD Project Principal Professional Experiences 


Length of time worked in the 


17.6 
field of education er 


Length of time as a principal 8.5 years 


PRINCIPAL CERTIFICATION AND PROFESSIONAL DEVELOPMENT 

All MLD Project school principals have a school administrator/principal certification. A 
total of 8 principals have a Master’s degree in Education and the remaining 4 have an Educational 
Specialist degree. All indicated that their major field of study for their highest degree was in 
education (i.e., educational leadership, instructional leadership, school administration). When 
asked what teaching certification they had received, they were split between elementary, 
secondary, and middle grades (Table 5). Eleven of the twelve principals had prior experience as an 
assistant principal; eight of the twelve had been a middle school teacher as well (average was 10 
years as a teacher). 
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Table 5. MLD Project Principal Teaching Certification 


Teaching Certification Pena 
Elementary certification 7 
Secondary subject-matter certification 6 
Middle grades certification 1 
Middle grades endorsement 5 
Other (e.g., administrative) 6 


* Number of principals does not total 12 because principals may have selected more than one response. 


When asked about whether they had received any specialized training in the instruction 
and organizational needs of a middle school, either during pre-service education or during 
professional development after joining the education workforce, the largest number of principals 
received such training while working as a teacher or administrator (Table 6). 


Table 6. MLD Project Principal Middle School Training 


Specialized Training on Middle Schools | Average Amount* 
Pre-service coursework 1:7 
Certification coursework iL) 
Professional development 2.6 
Master’s degree Dei 
Doctoral degree 1.0 


* 1=None; 2=1-2 courses; 3=3-5 courses; 4=6 or more courses. 


PRINCIPAL RATINGS OF GRANT ACTIVITIES AND 


THEIR IMPACT “I have grown as an instructional leader. My 
When principals were asked about the coach provided opportunities for me to see 
impact of the grant on their school, there was strong leadership. Our teachers have been able 
collective agreement that it was helping to to go into other schools and bring back 
improve leadership at their school, both for activities and strategies that work with our 
themselves as well as for their faculty (Table 7). students. Our school environment has 
Principals also overwhelmingly agreed that the improved.” - MLD Principal 
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improvements they made and the new practices they had implemented would continue after the 
grant ends. 


Table 7. MLD Project Principal Reports of Impact of MLD Project 


There is collective support among the faculty for the implementation of the 
MLD Project. 


4.2 


There is strong commitment among the faculty to support the leadership of 


my school. = 


Working with the leadership coach helped the faculty to improve their 
leadership. 


The MLD Project has improved student learning/achievement at my school. 
The improvements our school has made during the MLD Project will continue 4.5 
after the grant ends. ; 


* 1=Strongly disagree; 2=Disagree; 3=Neither agree nor disagree; 4=Agree; 5=Strongly agree. 


4.2 


When principals were asked about their 
level of satisfaction with the MLD Project services 
and activities, they indicated they were most 
satisfied with the leadership coach working with 
them, the leadership coach working with their 
leadership team, and hosting a mock STW visit at 
their school (Table 8). 


“The grant allowed for the implementation of 
distributive leadership. Through this, we were 
able to develop a common vision throughout 
our faculty about what good instruction looks 
like, sounds like, and how we want our 
classrooms to function. We changed our 


professional development model to one where 


we learn from each other. This allowed a team 
of leaders to be developed around the school. 
We truly have become a faculty of leaders.” 

- MLD Principal 
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Table 8. MLD Project Principal Satisfaction with MLD Project 


w 
Ne) 


Leadership coach working with me. 


w 
N 


Leadership coach working with teachers. 


w 
N 


STW Rubric annual data report. 


w 
uN 


Networking with other schools. 


w 
ou 


Visits to other schools. 


w 
N 


VAL-ED leadership assessment data report. 


* 1=Not satisfied; 2=Somewhat satisfied; 3=Satisfied; 4=Very Satisfied. 


“The overall impact of the grant has been very When principals were asked about the 
barriers they encountered when 


implementing the MLD Project activities, 


positive. It has given us purpose, direction, and 


guidance to improve what we do for the overall 
growth of the whole student.” there were no significant barriers reported. 


- MLD Principal The two items that were indicated to be 


minor barriers included lack of time 
necessary for adequate planning/implementation and lack of clear expectations about what is 
involved in the project. 
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Implementation Findings 


The purpose of the implementation study was to document the fidelity of implementation 
of the MLD Project at each treatment school, assessing whether the intervention was implemented 
as intended. 


FIDELITY MATRIX 

Using the MLD Project logic model, a measurement tool to track fidelity of implementation 
was developed (See Appendix A). The major project components were identified, as well as the 
key indicators to measure each major component. Then, for each indicator, an operational 
definition was developed, a data source identified, data collection schedule set, and an 
implementation score was determined so that an assessment of levels could be made (low, 
medium, high) for each indicator as well as across indicators for each major project component. 
Implementation fidelity was measured every year of the project (3 times), however, due to 
adjustments to project components after Year 1 of the project, only Year 2 and Year 3 
implementation fidelity was used and analyzed. 


The following data sources were used to assess fidelity of implementation: Coach’s Log; 
state leadership team activity reports, vision statement documents, Self-Study Teacher Survey; 
STW Rubric; improvement plans and action plans; meeting minutes; phone call logs; Edmodo 
software records; sign-in and meeting attendance lists; and leadership team meeting notes. 


Overall, each key project component was scored using multiple indicators. An 
implementation score was determined for each indicator. Some indicators were scored with a 
range of 0 to 1 where 0 is not completed and 1 is completed, while others had a range of 0 to 2 
where 0 is low implementation, 1 is medium implementation, and 2 is high implementation. All 
indicators except one were measured at the school level. The one indicator not measured at the 
school level was measured at the teacher level instead. Once the indicator level scores were 
calculated, then for each project component, the individual indicator scores were scored at the key 
project component level to arrive at a component level implementation score. For the component 
level score, the approach was to total up the scores on each indicator and apply a component level 
score of 0 to 2 (0 is low implementation, 1 is medium implementation, 2 is high implementation). 


Finally, for each component, a threshold was established for assessing fidelity of 
implementation across the sample (e.g., 85% percent of schools must have high implementation) 
in order to determine if fidelity of each component was met (Table 9). 
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The following key project components were measured: 


™ Create a powerful vision for high performance using the STW criteria; 

Engage in assessment and planning to identify needs, develop goals, and implement an 
action plan; 

MLD leadership coach works with leadership team; 

Principal mentor works with principal; 

Network school works with school; 

School engages in a leadership professional learning community; 

School participates in networking experiences; and 


School receives regular and systematic professional development trainings. 


Table 9. MLD Project Implementation Components and Thresholds 


Key Programmatic Implementation Threshold for Fidelity of 
Data Source(s) * : 

Components Score Implementation for the Sample 
Create a powerful Coach's Log, Online STW |} ow = 0-2 Ha desise 7006 Gt schiadls: 
vision for high Rubric data collection : . ona . 

: Medium = 3-7 Year 2 has medium or high implementation, 

performance using system, Self-Study ich= eal eC Sees sail 
the STW criteria Teacher Survey igh = ear 3 has high implementation 
Engage in 
assessment and = 

lanning to identi setae At least 70% of schools have high 
planning to identify Goachsilog Median = 2-7 t least 0% of schools have hig 
needs, develop goals, ; implementation 

: High = 8-9 
and implement an 
action plan 
MLD leadershi Low = 0-2 

a State team activity ; At least 70% of schools have high 
coach works with ; Medium = 3-9 : . 
é reports, Coach’s Log : implementation 
leadership team High = 10-11 
oe Low = 0 
Principal mentor State team activity ; At least 70% of schools have high 
; ees j Medium = 2 : : 
works with principal reports, Coach’s Log eg implementation 
igh = 
Low = 0 
Network school State team activity ; At least 70% of schools have medium or 
: Medium = 2 pe i : 
works with school reports ; high implementation 
High = 4 
School engages in a Sy 
. aS Low = 0-2 . 
leadership State team activity , At least 80% of schools have high 
; : Medium = 3-5 ; ‘ 
professional learning reports ; implementation 
: High = 6-7 
community 
School participates : = 
: P ‘ 'P State team activity Year 2 how = 0; At least 70% of schools have high 
in networking Medium = 1-2, . . 
: reports aoehe implementation 
experiences High = 3-4 
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Year 3: Low =0, 
Medium = 1-2, 
High = 3 


IMPLEMENTATION RESULTS 

The research question guiding the implementation study: Is the MLD Project intervention 
implemented with fidelity at each treatment school? An examination of implementation fidelity 
across the entire sample for each of the eight key programmatic components highlighted stable 
implementation of the majority of the key components (Table 10). In Year 2, seven of the eight 
components were implemented with fidelity. The only component in Year 2 that was not met with 
fidelity was regular professional development trainings. In Year 3, six of the eight components 
were implemented with fidelity. In Year 3, the two components that were not met with fidelity 
included the leadership coach working with the leadership team, and the school participating in 
networking experiences. In focus group interviews with coaches, they commented that the 
majority of schools implemented all of the key project components with fidelity with only one or 
two schools having a low level of implementation of a few components. 


Table 10. MLD Project Yearly Implementation with Fidelity by Component 


Create a powerful vision for high 
performance using the STW criteria 


MLD leadership coach works with 
leadership team 
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Principal mentor works with 70% 92% 75% 
principal Yes Yes 
Network school works with school 70% ek oe 

Yes Yes 
School engages in a leadership 80% 92% 100% 
| professional learning community Yes Yes 
School participates in networking 70% 100% 50% 
experiences Yes No 
School receives regular and 67% 75% 
systematic professional development 70% No Yes 
trainings 


As mentioned previously, the MLD Project was implemented in multiple districts in 
Kentucky and Michigan by the National Forum affiliates (Kentucky Middle School Association- 
KMSA and Institute for Excellence in Education-IEE). Each state had varying requirements and 
priorities and the affiliates had established infrastructures for overseeing services to the schools. 
As aresult of these differences, implementation of the networking and professional development 
activities of the project was approached in the same way that the National Forum uses the STW 
Rubric - by adapting to meet the needs of the schools involved and the realities of each setting. In 
Michigan, IEE used their central office location to host quarterly networking seminars for MLD 
Project schools. The principal and leadership team from each school attended the seminars where 
they participated in several key project components, including working with their leadership 
coach, working with their principal mentor, networking with other schools, and receiving 
professional development. It was an efficient way to implement the project activities while 
maximizing interactions among school personnel and the project team. In Kentucky, due longer 
distances between project schools which prohibited regular travel for networking meetings and 
professional development, project schools participated in project components at their individual 
school buildings with their leadership coach or other professional development trainer, and 
networking occurred when schools went on STW visits to other schools. All the schools across 
both states received the same services, however, the mechanism for delivering them varied. There 
were advantages to each delivery method. In Michigan, schools benefitted from more networking 
experiences. In Kentucky, schools received more individualized services at their buildings. 


An examination of implementation results by school highlight several key findings related 
to the variability of implementation. The table below (Table 11) shows the level of implementation 
of each key project component by year and it also shows which components were determined by 
the project leaders to be weighted as most important to the achieve. The first observation is that 
overall, implementation was relatively high across all components at all schools. It is clear from 
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these data that four schools (School 4, School 5, School 8, and School 12) had slightly higher and 
more consistent overall implementation than the other schools. We hypothesize that 
implementation in these four schools may be higher because they had a higher level of readiness 
for making improvements at the start of the project and their district was highly involved and 
supportive of the project and its goals. Although overall implementation was moderately high, the 
four most highly implemented schools were examined to determine whether their level of 
implementation resulted in higher intermediate or long-term outcomes, and this was not the case. 
While these four schools had slightly higher implementation, it did not manifest in higher 

outcomes at this time. It may be the case that more time with the higher level of implementation in 
place is necessary for it to impact outcomes positively. 


Table 11. MLD Project Yearly and Weighted Implementation by State and School 


Key Programmatic Components 


Year 2/Year 3 Level of Implementation (High, Medium, Low) 


States | Schools 
Vision Coach Visits PLC NW PD 
School 1 H/M 
School 2 M/H | H/H | H/H | H/H H/t | | H/M | 
School 3 H/M 
Pa | wa [wm | nn 


KY 
School 4 


School 5 
School 6 
School 7 
School 8 
School 9 
School 10 
School 11 
School 12 


MI 


20H 17H 
| | Highest Implementation (all high or all but 1 high) 


a Lowest Implementation (5, 6, or 8 medium; 2 low) 


Medium Implementation (3 or 4 medium) 
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Intermediate Outcome Findings 


STW RUBRIC “Our intentionality over the past two years in 
An analysis of the yearly aggregated STW working with the STW Rubric as well as being 
Rubric data using independent sample t-tests proactive instead of reactive, has helped us 


showed that there was statistically significant 


to all get on the same page and remain true 


: P . : to our vision. We’re very focused now.” 
improvement in staff ratings of their ae 
- MLD Principal 


implementation of each of the four key STW 
Rubric criteria from Year 1 to Year 3 of the MLD Project (Figure 2). Overall, staff reported 
significantly higher implementation of the STW criteria (p < .001) from Year 1 to Year 3. 


Figure 2. MLD Project STW Rubric Results by Year 


Average Staff Ratings on STW Rubric 


Academic Excellence Developmental Social Equity Organizational 
Responsiveness Structures 


'42014/15 ™2015/16 m™ 2016/17 


4=High quality; 3=Good quality; 2=Fair quality; 1=Poor quality ps.001 


SELF-STUDY TEACHER SURVEY 

A series of independent samples t-tests were run to determine if there were differences in 
the Self-Study Teacher Survey constructs from Year 1 to Year 3 of the MLD Project. Teachers 
reported statistically significant improvements in the following constructs related to 
collaboration, shared leadership, and classroom instructional practices. 


Team Practices increased - Teachers reported engaging more frequently in planning and 
coordination activities, curriculum integration, coordination of student assignments and 
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assessments, and contract with parents (Figure 3). These improvements were statistically higher 
in Year 3 than in Year 1 (p s .05). 


Figure 3. MLD Project School Improvement Self-Study Teacher Survey: Team Practices by Year 


Average Teacher Ratings of Team Practices 


4 
4 2014/15 

3 i 2015/16 
2 ™@ 2016/17 
1 a 
0 = 

*Planning and *Curriculum *Coordination of *Parent Contact 

Coordination Integration Student 

Assignments 


7=Daily; 6=Weekly; 5=Monthly; 4=Quarterly, 3=Several times a year, 2=Once a year; 1=Never ps.05 


Quality of Team Interactions Improved - Cohesion and harmony among teachers working 
together improved, teachers felt more prepared to work together in a collaborative way, and 
teachers agreed that they not only address student needs but have consistently high expectations 
for students (Figure 4). These improvements were statistically higher in Year 3 (p < .05). 


Figure 4. MLD Project School Improvement Self-Study Teacher Survey: Quality of Team Interactions by Year 


Average Teacher Ratings of Quality of Team 
Interactions 
5 = 


42014/15 
2015/16 
2016/17 


*Student Needs *TeamCohesion *Readinessfor *High Expectations 
and Harmony Team Practices _ for All Students 


5=Strongly agree; 4=Agree; 3=Neither agree nor disagree; 2=Disagree, 1=Strongly disagree ps.05 
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Team Decision Making Increased - Teachers reported increased decision making opportunities 
among their teams regarding team practices, school-wide policies and practices, and student 
performance and assessments (Figure 5). These improvements were statistically higher in Year 3 
(p s.05). 


Figure 5. MLD Project School Improvement Self-Study Teacher Survey: Team Decision Making by Year 


Average Teacher Ratings of Team Decision Making 


4 _ 

3 
'4 2014/15 

3 2015/16 
m 2016/17 

1 

0 a 

Team Practices School-wide Policies and Student Performance 
Practices and Assessments 
5=Very much; 4=Much; 3=Moderate amount; 2=Little; 1=Very little ps.05 


Teacher Efficacy and Collective Teacher Efficacy Improved - Teachers reported a statistically 
significant (p 2 .05) improvement in their individual, as well as collective, commitment to the 
success of their students by Year 3 of the grant. In other words, teachers reported stronger beliefs 
that they and their colleagues have what it takes to get students to learn, are able to get through to 
difficult students, are confident they will be able to motivate their students, and believe that if a 
student does not learn something the first time, teachers will try another way. 


Teachers report significant increases in the use of “best” middle-grades instructional 
practices (Figure 6). Teachers report engaging more frequently in the following instructional 
practices: 


e Small group active instruction 


e Integration and interdisciplinary practices 
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¢ Citizenship and social competence “The STW Rubric helps with ownership of our 
instruction improvement goals. It helps with the impetus 


© Critical thinking practices for the instructional shifts that need to happen. 


That focus for the principal then being able to 
e Reading skill practices be proactive, not trying to always catch up, but 
leading. Truly leading.” 
- MLD Leadership Coach 


e Writing skill practices 


e Listening and verbal skill practices 


Figure 6. MLD Project School Improvement Self-Study Teacher Survey: Classroom Practices by Year 


Average Teacher Ratings of Classroom Practices 


42014/15 
2015/16 
2016/17 


*SmallGroup “Integration Authentic *Citizenship *Critical Math Skills vee “Writing Skills vee 
Instruction Assessment Thinking 


‘~ 


w 


nN 


o 


7=Daily; 6=Several times a week; 5=Weekly; 4=Several times a month, 3=Monthly, 2=Several times a year; 1=Never ps.05 


COACH’S LOG 

The 12 MLD Project schools in Kentucky and Michigan received coaching services from a 
leadership coach from fall 2014 through spring 2017. The leadership coaches completed an online 
Coach’s Log for any key or substantive visit, conference call, email communication, or webinar 
with their school. Overall, coaching services were a very intensive and ongoing aspect of the MLD 
Project. Throughout the three-year project, leadership coaches made 715 contacts with project 
schools consisting of 1,103 different activities for more than 3,600 hours (Table 12). This 
amounted to an average of 59 contacts per project school. 
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Table 12. MLD Project Coaching Contacts and Activities by Year 


Coaching Services 
Annual Contacts and 

Activities Total 
Number of schools served 12 
Number of contacts 715 
Number of hours 3,687 
Number of activities 1,103 
Average contacts per school 59 
Average hours per contact 5.1 


Although there were some key elements of coaching that were ongoing throughout all 
three years of the MLD Project (i.e., providing generalized coaching services; conducting 
walkthroughs and observations; and working with schools to use data effectively for decision 
making), there were other coaching services that evolved in response to the progress and needs of 
the school (Table 13). 


Table 13. MLD Project Key Coaching Elements by Year & Descriptions 


Key Coaching Elements 
Year 1 (2014/15) Year 2 (2015/16) Year 3 (2016/17) 

Key Elements in Year 1 
e Developing trusting 

relationships; and 

e Learning about the school 
Key Elements in Year 1, Year 2, and Year 3 
e Providing generalized coaching services (e.g., mentoring, providing resources); 
e Conducting walkthroughs and observations; and 
e Working with schools to use data effectively for decision making 


Key Elements in Year 2 and Year 3 

e Capitalizing on networking opportunities (e.g., state 
workshops, annual STW Conference); 

Utilizing STW visits to drive change; 

Utilizing mock STW visits to improve teaching practices; 
Assessing school readiness to apply for STW designation; and 
Focusing on informed and fluid RTI Tiered services 


Key Element in Year 3 
e Planning for sustainability; and 
e Applying for STW designation 
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In Year 1, coaches focused their efforts on developing trusting relationships, as well as, 
learning about their schools in order to provide more responsive coaching. Starting in Year 2 and 
continuing into Year 3, coaches reported capitalizing on network opportunities; utilizing STW 
visits to drive change; utilizing mock STW visits to improve teaching practices; assessing their 
readiness to apply for STW designation; and focusing on informed and fluid RTI Tiered services. In 
Year 3, coaches focused on planning for sustainability with the end of the grant approaching. 
Finally, although schools explored their “readiness” in Year 2, it was not until Year 3 that schools 
were applying for STW designation with their coach’s assistance. The key coaching elements are 
further described in the section below. 


Descriptions of Key Coaching Elements 


e Developing trusting relationships with the principal, leadership team, and teachers; 
e Learning about the school in order to prioritize needs and provide responsive coaching; 


e Providing generalized coaching services, such as coaching conversations and mentoring; 
participating in team (e.g., administration, leadership, PLCs) meetings; assisting with state and 
district requirements; helping with the creation of the master schedule; advising on the 
recruitment and hiring of staff; attending school events; and providing resources and 
professional development to the principal, teachers, and school; 


e Conducting walkthroughs and observations (e.g. classrooms, PLCs, school) utilizing 
protocols or observation tools (e.g., STW Rubric, school district instrument) with a focus (e.g., 
determining professional development needs and levels of engagement, assessing progress on the 
STW Rubric and best instructional practices); debriefing/calibrating with an administrator or 
the leadership team; providing feedback and strategies to teachers; shadowing principals and 
assistant principals to observe their interactions with staff and students; and facilitating peer 
observations; 


e Working with schools to use data effectively for decision making by utilizing multiple data 
sources (e.g., STW Rubric, Teacher Survey, VAL-ED Assessment, state achievement scores, mock 
visit, walkthrough/classroom observations, discipline and referral data), disaggregations, and 
longitudinal data for: facilitating the principal’s reflection on their practice and the 
development of a professional growth plan; utilizing formative assessment data to impact 
teacher instructional practices and address student learning styles; establishing and adjusting 
RTI tiered services; monitoring early indicators to better support students; informing (and 
adjusting) student discipline policies and programs; aligning state and district expectations, 
initiatives, and action plans with the school’s improvement plan; and celebrating successes; 
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e Capitalizing on networking opportunities (i.e. STW Conference, networking meetings, STW 
site visit training) by sharing successes and resources from other coaches, project schools, and 
STW schools; debriefing after sessions and discussing how to implement best practices and 
other takeaways at their school; building capacity through opportunities to practice skills (e.g., 
collaboration skills) during networking meetings; and learning more about the STW process 
through participation in site visit trainings; 


e Utilizing STW visits to drive change by determining the school visit focus then selecting the 
relevant school and teachers to visit; discussing observations and questions with the visited 
school teachers and administration; collaboratively debriefing and discussing takeaways; 
communicating these observed best practices and programs to the rest of the school; and 
adapting/implementing practices and programs to meet the needs of their school; 


e Utilizing mock STW visits to improve teaching practices by monitoring instruction more 
deeply with the STW Rubric in preparation of the visit and the visiting team’s expectations; 
debriefing with the visiting team’s constructive feedback; validating teaching practices and 
confirming administrator assessments; and discussing “applause” and “concerns” to direct 
further instructional changes; 


e Assessing school readiness to apply for STW designation by reviewing the application, 
explaining the process, and identifying the necessary data to prepare; gathering information 
at the STW Conference and through participation in STW visits and a STW site visit training; 
meeting with teams and individual stakeholders (e.g., principal, administrative team, 
leadership team, teacher(s), central office) to assess their perceptions, concerns, and support of 
the application process and timeline; and either deciding to apply during Year 3, or making 
plans to apply in the following year; 


e Focusing on informed and fluid RTI tiered services by utilizing data at the beginning, 
middle, and end of the year to inform student placement; analyzing areas of weakness to 
develop lessons for mastery and novice reduction in math, ELA, and ELL for individual 
students; re-teaching standards with lowest scores; and looking at teacher evaluation data 
and certifications to refine both the master schedule focusing on RTI time, as well as needed 
teacher professional development; 


e Planning for sustainability by introducing the sustainability protocol and matrix (i.e., 
assigning initiatives to quadrants, reflecting on priorities, and creating an action plan); 
planning to support ongoing practices for next year; understanding the “why” instead of just 
focusing on the “what” and “how’; “ramping up” distributive leadership; “institutionalizing” 
changes to “survive” staffing changes—especially the principal and superintendent; 


developing an orientation for new faculty on the school’s culture and way of doing things; 
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taking on roles and responsibilities that the leadership coach was providing; utilizing the STW 
Rubric for reflecting on strengths, challenges, and next steps in sustaining best practices; and 
focusing on mediating questions from mock visits and pre-visits to refine practices and 
consistency throughout school; 


e Applying for STW designation by assessing school readiness to apply; engaging with the 
administrative team, leadership team and faculty in collaboratively gathering evidence to 
assess and align their school with the STW Rubric, as well as make any needed changes to 
strengthen their school’s use of the STW criteria; examining and disaggregating achievement 
data to discuss how to make meaningful changes to reach gap students; explaining the 
application process and STW designation to parents, school board, and the community; 
reflecting on multiple iterations of the application before submitting an application; visiting 
individual classrooms to make specific recommendations to teachers; conducting school 
walkthroughs to measure school readiness for the designation visit; and hosting the visiting 
designation team. 


The journey continued after their successful STW designation with much celebrating by staff, 
students, and stakeholders (e.g., school district, school board, politicians) and the presentation 
of the STW banner—or multiple presentations of the STW banner with different audiences. 
Afterwards schools reported: being ready for new initiatives; reviewing exit report notes; 
having open and frank conversations about growth areas in order to determine what to 
highlight in their STW School presentation at the annual conference; presenting to the school 
board; hosting their first non-project school for a visit; and celebrating again at the STW 
Conference. 


FOCUS GROUPS 

Qualitative data analysis of the principal and leadership coach focus groups that occurred 
after each year of the MLD Project provided formative data on the quality and fidelity of 
implementation, as well as the change process. The meaningful content of the qualitative data 
were examined using an inductive approach. Recurrent themes were identified with a focus on 
whether the content represented group-shared ideas. Through this content analysis, the data were 
summarized, with the highlights below: 


Year 1 Findings: 


e Coach’s successful process to build trust, credibility, and rapport with principal 
and staff to facilitate responsive coaching; 
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Coach’s ongoing process to mentor and model a collaborative, distributive 
leadership style for the principal; 

Development of teacher leaders through the creation of leadership teams; 
Utilization of project data (i.e., STW Rubric, Self-Study Teacher Survey, VAL-ED) in 
a purposeful manner to secure buy-in from staff for implementation of the MLD 
Project; 

Identification of goals and challenges for implementing the project in Year 2; and 
Coach’s assessment that the implementation of the MLD Project had positively 
impacted school culture. 


Year 2 Findings: 


School’s focus on developmental responsiveness and student needs in Year 2 to 
improve school climate and culture—most frequently through the adoption of an 
advisory program; 

Improvements in collaborative leadership owing to principals creating a 
supportive risk-taking environment and intentionally growing teacher leaders 
with increased decision-making power and autonomy; 

Positive impact of developing principal and teacher leadership through 
professional development and networking opportunities through the MLD Project; 
Improving the principal’s leadership skills through reflective conversations with 
the coach on the VAL-ED data to identify strengths and weaknesses in order to 
develop a professional growth plan with targeted goals; 

Process of developing teachers as instructional leaders, and building principal 
capacity to support teachers in the classroom; and 

Plans to build capacity for sustainability beyond the grant. 


Year 3 Findings: 


Key ingredients for replicating success in transforming project schools into 
nationally recognized STW Schools; 

Descriptions of unexpected impacts of participation in the MLD Project that 
positively impacted project schools; 

Principal assessment that their school’s collaborative leadership has improved as 
a result of participation in the MLD Project; 

Importance of building school culture focused on the holistic child with a student- 
centered environment, which principals and coaches believed improved 
academics; 

Principal assessment that developing teacher leaders for a collaborative, 
distributive leadership model is the key to building capacity for sustainability 
beyond the grant or principal turnover; and 

Recommendations for the implementing the MLD Project in future work. 
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Final Outcome Findings 


PRINCIPAL EFFECTIVENESS 

The final outcome of principal effectiveness was measured with the VAL-ED Leadership 
Assessment. VAL-ED provides scores for six core components of leadership: high standards for 
student learning; rigorous curriculum; quality instruction; culture of learning and professional 
behavior; connections to external communities; and performance accountability. VAL-ED further 
breaks down six key leadership processes (planning, implementing, supporting, advocating, 
communicating, and monitoring) within each core component to provide an in-depth picture of 
each principal’s leadership. At the project level, principals improved for each core component and 
key process from Year 1 to Year 3 of the MLD Project, with principals demonstrating the greatest 
growth in the areas of culture of learning and professional behavior, quality instruction, 
performance accountability, and high standards for student learning (Table 14). 


“My superintendent asked, ‘You sure you 
want to have your staff take the VAL-ED?’ 
Yeah, I do. | mean this is my community. 
These are the people I work with, and if they 
see me doing something that could become 
better, why not become better? So that’s 
check your ego at the door and just do what 
you need to do.” 


- MLD Principal 


Leadership skills and behaviors also improved as measured by the principal’s overall 
effectiveness score, which is indicated by both a mean score and a percentile score. The mean 
score is the average of all respondent groups (i.e., principal, teachers, supervisor) on all key 
processes and core components to determine the individual principal's performance level. VAL- 
ED assigns performance levels (i.e., distinguished, proficient, basic, below basic) according to set 
ranges of mean scores that do not change based on the national sample. The number of schools 
with proficient or distinguished principals grew from two schools in Year 1 to nine schools by the 
end of Year 3 (Table 15). The percentile score is the principal’s mean score in comparison to a 
national sample of principal means. Based on the percentile score, twelve principals grew as 
leaders when compared to the national sample. 
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Table 14. MLD Project VAL-ED Principal Effectiveness Scores by Year 


Average Principal Effectiveness Scores on Leadership Core Components and Key Processes | 


Year 1 (2014/15) 


Year 2 (2015/16) | Year3(2016/17) | 


High Standards for Student Learning 


Implementing 
Supporting 
Advocating 
Communicating 


Planning 3.44 3.53 
Implementing 3.36 

Supporting 3.48 

Advocating 3.21 

Communicating 3.23 

Monitoring 3.48 

Rigorous Curriculum 

Planning 3.14 


3.17 


3.43 


Monitoring 


Quality Instruction 


Planning 


Implementing 
Supporting 
Advocating 


3.39 


Communicating 


3.36 


Monitoring 


3.49 


Culture of Learning and Professional Behavior 


Planning 3.59 

Implementing 

Supporting 

Advocating 3.43 

Communicating 3.28 

Monitoring 3.14 

Connections to External Communities 

Planning 2.98 3.31 
Implementing 2eo2. 3.53 
Supporting 2.93 3.59 
Advocating 2.80 3.49 
Communicating 3.33 3.55 
Monitoring 3.03 3.33 
Performance Accountability 

Planning 3.23 

Implementing 3.28 
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Supporting 
Advocating 
Communicating 
Monitoring 
VAL-ED Performance Levels 
Below Basic 
(1.00-3.28) 


Table 15. MLD Project VAL-ED Principal Overall Effectiveness Scores and Performance Levels by Year and School 


Principal Overall Effectiveness Scores and Performance Levels 
Mean Score Percentile Score 


| Year2 | Year3 | Change | Year1 | Year2 | Year3 | Change | 


School * 


fs] oe [om [a [| 
fa] «> [om [oe [am [am 
[os [| ow [om [om [wm [oe 
a 
[os [ae [ss [as [om [mm [ae 
a [sae [ox [on [om [om 
fos faa w= fan [os [am [a 
Ga = [om [aw [sss 
Pele [a [am [ae [am 


18.2% =. 30.1% 11.9% 


School 3 


School 4 


School 5 


School 7 


School 8 


School 9 


School 10 


School 11 


School 12 16.8% 37.4% 


VAL-ED Performance Level: 


principal turnover |Below Basic (1.00-3.28 
during the 3-year 


MLD Project. 


School 13 


*Two schools had 


Proficient (3.60-3.99) 
Distinguised (4.00-5.00 
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STUDENT ACHIEVEMENT TEST SCORES 

The final outcome of student achievement was measured with the state achievement tests: 
the Kentucky Performance Rating for Educational Progress (K-PREP); and the Michigan Student 
Test of Educational Progress (M-STEP) test. Each test was administered annually in 2015, 2016, 
and 2017 at all project and comparison schools. 


Overall school achievement 

Overall, in the state of Kentucky, student achievement on the K-PREP test in 2017 
increased slightly at the middle school level with achievement gaps between different groups of 
students persisting in many areas. The percentage of middle school students performing at the 
Proficient/Distinguished levels on the KPREP test increased in reading, social studies, and 
language mechanics. A comparison of the MLD project schools to the state at the 8th grade level 
showed that two schools had larger reading growth than the state and two schools had larger 
math growth than the state (Table 16). See Appendix B for the full table containing all grade levels. 


Table 16. MLD Project K-PREP Achievement for Overall State and 8 Grade 
Project Schools by Year 


Reading K-PREP Results 
a 2017 


% of Proficient or Advanced Growth 


2015 to 2017 
ee 
om | 08 


50.9% -5.0% 


43.4% | 51.5% | 


lia | 48.0% | 47.3% -10.3% 


Renae 2035-2017 a pie % of Proficient or Advanced Growth 


2015 to 2017 
2016 2017 
a 
Overall State 44.2% 45.5% 48.7% 
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School 1 42.9% 40.7% -6.1% 
44.0% 34.0% -3.4% 


24.1% 27.6% | ( 
31.5% 35.7% 


38.7% 1.9% 
|_| Higher than the overall state 


Overall in the state of Michigan, student achievement on the M-STEP achievement test in 


2017 increased for mathematics in the middle grades of 6, 7 and 8. The Michigan Department of 
Education, however, noted “the English language arts scores are disappointing” in that ELA scores 
decreased in 2017 in grades except grade 5. A comparison of the MLD project schools to the state 
at the 8th grade (note that school level results are not available) showed that five schools had 
larger ELA growth than the state and one school had larger mathematics growth than the state 
(Table 17). See Appendix B for the complete table containing all grade levels. 


Table 17. MLD Project M-STEP Achievement for Overall State and 8 Grade 
Project Schools by Year 


ELA M-STEP Results - 
2015-2017 % of Proficient or Advanced Growth 
2015 to 2017 


Pp —CSCid:«C205 | 2016 | 2017 
eee CS 
=< Tas Ts Tan | oa 


School 7 
School 8 


ee 
sii ro | 


EEE 
Mathematics M-STEP % of Proficient or Advanced Growth 
Results 2015-2017 

2015 to 2017 


P20 | 2036 | 2017_| 
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a 
== Tas Par Pos [a 


School 8 


School 10 


— Higher than the overall state 


Impact study final analytic sample 


As described in the Evaluation Design section, the impact study tracked a cohort of project 
and comparison students starting in 6' grade in the 2014/15 school year over three years by 
matching individual students over time. The final analytic sample for ELA/reading included 1,300 
intervention students and 4,412 comparison students. The final analytic sample for math included 
1,304 intervention students and 4,433 comparison students. Students were from a total of 50 
schools (12 intervention and 38 comparison). Demographics of the sample are in Table 18. 


Table 18. MLD Project Demographics of Final Impact Study Analytic Sample 
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English No 1,279 98.1% 4,364 98.4% 
Language 

Yes 25 1.9% 
Learner Status 7 69 1.6% 


Impact study baseline equivalence 


Baseline equivalence was conducted using the final analytic sample. For confirmatory 
research question one, 5‘ grade math achievement scores were used, and for confirmatory 
research question two, 5 grade ELA/reading achievement scores, were used to examine the 
equivalence between treatment and comparison schools. A 2-level model, where students were 
nested within schools, was performed to estimate the treatment minus comparison difference as 
done in the impact analysis without including any other covariates into the model. The structure 
of the model is shown below: 


Level 1: 

Yij = Boj + &i 

Level 2: 

Bo; = Yoo + Yoi(Trt) + [oj 

Where 

Y,, =5th grade measure (ELA/reading or math) of the it student in the jth school. 
Trt = 1 ifin treatment schools, =0 if in comparison schools 

€ = residual error, €j ~N(0, 02) and po; ~N(0, Tt?) 


Fitting the above model to the data would produce an estimate of Yo1, which represents the 
estimated mean difference in the pre-test measures. 


For unadjusted means and standard deviations, descriptive statistics were computed for 
the treatment and comparison groups for both ELA/reading and math. Then, the standardized 
difference was calculated as the model adjusted mean difference divided by the pooled standard 
deviation using this formula - 


(n, -1)S; + (n, -1)S3 
(n, +n, —2) 


S nooled = 


Where niand n2 are the student sample sizes, and S; and S2 are the unadjusted standard 


deviations for the intervention and comparison group, respectively. 
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The standardized difference was then multiplied by a factor of w = [1 - 3/(4N - 9)], with N 
being the total sample size. Because of the large sample sizes, w gets close to one and as a result 
this correction does not make much difference of the original standardized difference. 


There were no significant differences in baseline means (Table 19). In other words, the 
intervention students and the comparison students were equivalent on achievement test scores 
prior to the start of the intervention. Therefore, any differences in achievement between the 
groups after the intervention can be considered a result of the intervention and not because the 
groups began at different levels of achievement. Although the final analytic sample did not show 
any baseline difference, since the standard deviation units exceeded .05, adjustments were made 
in the analyses at both the student and school levels. 


Table 19. MLD Project Baseline Balance Testing Results of Student Achievement Using Final Analytic Sample 


Intervention Group Comparison Group Standardized 
Model _ 
ainerea Difference 
Unadjusted | Student | Model- | Unadjusted | Student ae Effect Size w/ 
Standard Sample | adjusted Standard Sample i anh Hedges' 
Difference 


Deviation Size Mean Deviation Size Correction 


Outcome Unadjusted 
Domain Mean 


ELA/Reading 
- 5th grade 


0.9458 1,300 | -0.0089 0.9599 4,412 -0.068 


Math - 5th 
grade 


Impact study statistical models 


The final analytic sample, evaluating the impact of the intervention, consists of 8th grade 
achievement scores after following the cohort of 6 graders who continued to belong to the same 
treatment schools as stated in Evaluation Design. Any students with missing pre-test achievement 
scores were excluded from the analytic sample. In addition, any students who moved during the 
project years were not included in the analysis. For confirmatory research question 1, 8" grade 
state standardized math achievement scores were the primary outcome. For confirmatory 
research question 2, 8» grade state standardized ELA/reading achievement scores were the 
primary outcome. The impacts of the intervention were tested on the combined cohorts of 
Kentucky and Michigan. 


From the Unconditional Means Models, an ICC of 4.3% for ELA/reading and an ICC of 8.2% 
for math suggested the use of Hierarchical Liner Models (HLMs). A series of 2-level models, where 
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students were nested within schools, were executed to analyze the intervention effect on student 
achievement. 


Let i refer to students and j refer to schools. Let ¥;; be 8» grade achievement (math or 
ELA/reading) for student i in school j. The model used to estimate the impact for a particular 
contrast has the two-level structure of students nested in schools, as shown below: 


Level 1: 
K 
Y,= Bo; + Bi, Vores) + y Bex (Xj) + é; 
kal 


Level 2: 
Boj = Yoo + You(Trt) +} Yo; (wij) + Hoj 
Where 
Y = 8" grade achievement scores (ELA/reading or math) of the i” student in the j'" school. 
Ypre.ij = 5" grade achievement scores (ELA/reading or math) of the i student in the j'" school. 
Xx represents k=1, 2, ..., K student level covariates included in the model. 
wjrepresents j=1, 2, ..., J school level covariates included in the model. 
Trt = 1 ifin treatment schools, =0 if in comparison schools 
€j = residual error, €jVN(0, 0) and [o;~N(O, t’) and independent of €;. 


Fitting the above model to the data would produce an estimate 7,, that represents the 


estimated impact of treatment. 


The following student and school-level covariates were used in the analyses: 


Student-level covariates — 5th grade pre-test achievement scores, race/ethnicity (white vs. 
not-white, African American vs. not African American, Hispanic vs. non-Hispanic), gender, LEP 
status, free/reduced lunch status, and special education status; and 


School-level covariates - free/reduced lunch, enrollment, NCLB AYP status, and school 
demographics. 
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Other than the baseline 5 grade achievement scores, the other covariates did not change 
much of student and school level variances. However, all possible covariates were included as part 
of adjusting for any baseline differences with the goal of improving the accuracy of the coefficients 
and ensuring alignment with the covariates used in the propensity scores while matching 
comparison schools. STATA xtmixed was used to execute the models. In addition, a correlation of 
0.69 to 0.71 was established between baseline 5th grade achievement scores and the 8" grade 
achievement outcome for both ELA/reading and math. 


The effect size was calculated using Hedges’s g ES index. It is defined as the standardized 
mean difference, which is the difference between the mean outcome of the treatment group and 
the mean outcome of the comparison group, divided by the pooled within-group standard 
deviation (SD) on 8 grade achievement scores. Since the achievement scores were at the student- 
level, the student-level standard deviation was used. 


The final models, adjusting for all possible covariates, suggest no significant intervention 
effect on either ELA/reading scores or math scores (Table 20). In other words, the students that 
received the MLD Project intervention performed the same on both the ELA/reading test 
(Appendix C) and the math test (Appendix D) as the comparison students after three years. 
Neither the p-values nor effect sizes suggest a significant intervention effect on achievement. 


Table 20. MLD Project Impact Study Results 


Intervention Group Comparison Group 
Impact Standard- 
Unadjusted Unadjusted Estimate | ized Effect Impact : 
pasa Cluster | Student 4 Cluster | Student 4 (Model Size w/ Standard ni 
abi Sample | Sample Standard | Sample | Sample Standard Adjusted Hedges’ Error Ce 
Size Size Mean | Deviation Size Size Mean Deviation Mean Correction 
Difference) 
ELA/ 
geading 12 1,300 | -0.0178 0.936 38 4,412 -0.0223 0.955 0.0431 0.045 0.0551 434 
Math 12 1,304 | -0.0898 0.877 38 4,433 -0.0169 0.935 -0.0005 -0.0005 0.0816 995 


Even though the impact analyses did not find an overall intervention effect on either 
ELA/reading scores or math scores, it should not be interpreted to mean that the MLD Project 
intervention was not effective, but that project schools may need additional time to fully 
implement and refine the model with middle grades students to advance achievement scores. We 
can further hypothesize that the positive changes noted in prior sections of this report on school 
organization and climate factors such as leadership, culture, climate, and instructional practices 
are a positive precursor to improvements in achievement since achievement does not improve 
without corresponding improvements in the teaching and learning environment. 
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Unexpected Findings/Spillover Effects 


While the MLD Project logic model (Figure 1) clearly illustrates the anticipated intermediate 
and long-term outcomes expected from the implementation of the eight programmatic 
components, in practice there were additional, unexpected outcomes as a result of the spillover 
effect that benefitted the project schools, state leadership teams, and the project leadership team. 
This section highlights the most significant findings. 


STW DESIGNATIONS 

Although the focus of the MLD Project was to lay a foundation for a school’s journey 
towards a National Forum STW designation, it was not an expectation before the end of the 
project. That 7 of the 12 project schools applied for and successfully received a STW designation 
before the end of the project (and another 3 schools during the no-cost extension), is a major 
triumph of the project and a tribute to the hard work of all of the stakeholders (i.e., administrators, 
faculty, students, and coaches) involved. Coaches believed that teachers and administrators had a 
“growth mindset” and an undefinable “readiness factor” to improve their schools. In addition to 
the support of the school district the following 5 areas of focus from the MLD Project were deemed 
essential to their success: 1) support from the leadership coach; 2) STW Rubric to focus school 
improvement; 3) transformational power of STW visits; 4) adoption of a distributive, collaborative 
leadership model; and 5) creation of a student-centered learning environment. 


MOCK STW VISITS 

Although the mock STW visits were not a planned program component, they were added 
in Years 2 and 3 of the MLD Project. The objective was to support project schools on the pathway 
towards a STW School designation after the grant ended. With this end goal in mind, the project 
team replicated key elements of the STW application process and school visit to provide project 
schools with a rich simulation, while supporting them with the expertise of their leadership coach 
and a national team of STW-trained middle-grades educators (see Lessons Learned section for 
more information about the team and process). The STW process and feedback provided the 
majority of schools with the confidence and tools needed to apply for STW designation. However, 
an added benefit was that principals, school leadership teams, and teachers indicated that the 
mock STW visits had the greatest impact on their improvement progress and goals. 


Although originally intended as a learning opportunity for the projects schools, the mock 
STW visits also greatly benefited the overall project team (i.e., project leadership teams, state 
leadership). By visiting each school, the project team became more familiar with the challenges 
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project schools were facing, the early successes they were experiencing, and identified additional 
supports that would benefit the project schools. Ultimately, the mock STW visits impacted the 
project team’s understanding of the project and informed future project-level discussions and 
decisions. 


DISSEMINATION OF THE MLD PROJECT BEST PRACTICES 


Even before STW designation, projects schools “People were asking all the time 
were being utilized as mentor schools hosting schools about our school! Directors of 
both from within their school district (including Instruction from other school districts 


elementary and high schools), and schools outside of wanted to come and see our school! 


Other schools in the district want to 


the district who wanted to visit and adopt the best ; 
adopt some of the things we’ve put 
/~ ao : in place.” 

addition, teachers and principals were presenting at - MLD Principal 


practices that MLD Project schools had to offer. In 


state and national level conferences on their school’s 


transformation due to their participation in the MLD 
Project. 
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Challenges 


With any complex project, such as the MLD Project (i.e., 8 unique programmatic components, 
multiple states, multiple districts), a variety of challenges were bound to occur. Many of challenges 
faced by the MLD Project leadership team, state leaders, leadership coaches, and principals were 
monitored, discussed, and resolved. Other challenges, however, were more tenacious and 
strategies were implemented to ameliorate barriers to implementation when they could not be 
removed. Data collected for the project as well as minutes from project team (i.e., leadership team, 
state leaders, and evaluators) phone calls and meetings highlight the most challenging barriers. 


EVOLUTION OF THE PRINCIPAL MENTORSHIP 

The principal mentor was one programmatic components of the MLD Project that was never 
fully implemented as intended. Originally, each project school principal was to be paired with a 
principal mentor, who would meet in person with them 6 times a year. However, it was challenging 
to secure enough principal mentors for a one-to-one match for each of the 12 principals, and the 
frequency of 6 visits a year was prohibitive due to geographic distances between schools and time 
constraints. In the end, each state provided mentoring experiences for MLD Project principals 
through the following substitutions: principal mentors worked with principals virtually via phone 
or email in addition to in-person visits; principals received mentoring during visits to other STW 
schools or on STW designation visits; and principal mentors met with groups of principals at 
state-sponsored events. Given these substitutions, the project also reduced the amount of required 
mentoring to 4 sessions per year. 


DISTRICT DEMANDS 

Unfortunately, district demands often took precedence, and sometimes attention, away from 
the MLD Project focus. In particular, principals were sometimes laden with district changes, issues, 
and requirements that reduced their autonomy and decision making as leaders. For example, 
breaking up the middle-grades program across two buildings with the 6 grade housed in the 
elementary school and the 7‘ and 8t grade housed in the high school, which hindered the 
creation of a single, unified leadership team; tearing down the middle school building and 
relocating students and faculty into the high school, which created a loss of identity; sharing 
teachers between the middle school and the high school, which overburdened the teachers in 2 
schools; removing a principal “effective immediately” with the superintendent assuming the role 
of interim principal; imposing districtwide responsibilities (i.e., Athletic Director, Assessment 
Director) in addition to the principal’s duties, which resulted in a conflict of interests; and needing 
to focus on meeting district demands, which led to the premature exit of the MLD Project for one 
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school. All of these challenges negatively impacted school culture. Leadership coaches employed 
responsive coaching to support the changing needs of schools during these transitions and to 
provide them with the resources that they needed. 


In addition, the districts had their own professional development requirements for school 
attendance, which sometimes resulted in competing agendas, as well as made it difficult for 
principals to attend MLD Project professional development and networking opportunities. One of 
the ways that leadership coaches responded was to help schools to align these competing services, 
and manage the many changes, by making connections with the school’s school improvement plan 
and the STW Rubric. This assisted many schools with staying focused on their vision for 
improvement and not becoming overwhelmed nor pulled in too many different directions. 
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Lessons Learned 


There are a myriad of lessons learned from the MLD Project that are important to 
document as part of the evaluation of the implementation and outcomes of the project. These 
lessons provide valuable information for future implementation of this project and serve to inform 
other middle-grades schools embarking on leadership improvements and the implementation of a 
collaborative leadership model. Although there were many differences between the twelve project 
schools, such as their location (rural/small town/suburban), enrollment, building grade 
configuration, and district and state requirements, they shared many of the same challenges and 
struggles to improve their leadership practices, develop a student-centered focus, and improve 
outcomes. The lessons presented below reflect findings from the data collected as part of the 
evaluation from all stakeholders in the project including principals, leadership teams, teachers, 
coaches, principal mentors, and state project leaders. 


VALUE OF PRINCIPALS WORKING WITH THEIR FACULTY TO ADOPT A GUIDING VISION 

The STW criteria provided a vision to MLD Project schools for what they wanted their schools 
to become. Using the STW criteria and rubric helped schools to delve deeper through articulating 
the meaning of each criterion, how they could implement it successfully by identifying their 
school’s strengths and weaknesses, and validating their existing best practices. 


A guiding vision: 


¢ Serves as a framework and a focus. 


“| attribute the improvements we’ve 
_ made to establishing a vision. A lot of 
B01ng: times you’ve got to be able to see the full 


¢ Provides a picture and pathway of where the school is 


picture of where you are going before 
you take the steps to get there.” 
¢ Serves as a basis for reflection and continuous - MLD Principal 


¢ Brings the faculty together with a shared purpose. 


improvement. 


¢ Keeps the focus on student needs and student-centered learning. 


IMPACTFULNESS OF USING THE STW RUBRIC FOR CONTINUOUS IMPROVEMENT 

Through the MLD Project, schools adopted and then facilitated their own continuous school 
improvement model where data was used at every stage of a never-ending cycle to: inform 
planning, set targeted goals, reflect on and evaluate progress, and refine implementation. The 
adoption of a continuous improvement structure began when schools were introduced to the STW 
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criteria as a vision and the STW Rubric as a tool to inform school decision making with a data- 
driven approach. This structured approach of using the STW Rubric in ongoing school 
improvement was critical to keep the focus of MLD Project schools on their goals and its existence 
will assist these schools in sustaining their work after the project ends. 


Key aspects of how the STW Rubric facilitated continuous improvement: 
¢ Best middle-grades practices based on evidence-based research. 


¢ Endorses best practices that are already occurring in the school while creating a systemic 
approach to encourage consistency of practice and expectations. 


¢ Provides a common language for 


“The rubric helped to show us what is best discussions about improvement leading to a 
for kids. And it was the evidence. It wasn’t 


; eal ate : shared vision. 

just this is what we feel is important. It is best 

practice, research based, so it helped in * Provides a structure for setting classroom 
getting the buy-in.” and building-level improvement goals 


- MLD Principal 


* Powerful tool for cultivating individual and 
school-wide reflective practices. 


¢ Empowers teachers to accept the challenge of high performance and strive to continue to 
improve. 


¢ Empowers teachers to focus on growing the “whole child” instead of a solitary emphasis on 
academic achievement. 


¢ Tool to unify multiple programs the schools is required to implement with the school’s shared 
vision to create one plan. 


¢ Defines the school’s identity as it was ingrained in the culture and informed every decision. 


IMPORTANCE OF COLLABORATIVE LEADERSHIP 

The MLD Project provided the structure and support to schools to facilitate the 
development of collaborative leadership. Project school principals now utilize a collaborative 
leadership style that empowers their teachers and provides them with opportunities for 
collaboration, reflection, and shared leadership through participation in the leadership team, 
interdisciplinary teams, PLCs, etc. More importantly, these collaborative leadership practices built 
school capacity to sustain the changes they were implementing. Time provided for collaboration, 
(e.g. PLCs, grade-level teams, interdisciplinary teams, etc.) often focused on the individual 
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student, such as monitoring early indicator data, goals, and challenges to identify ways to provide 
a support system for each student and to celebrate their successes. Project schools had begun 
“teaching smarter” as a result of their collaboration since they were sharing ideas, coordinating 


lessons, and looking more critically at their teaching practices. 
Collaborative leadership empowers teachers to: “The principal has really empowered his staff, 
and that’s when the school really started to take 
off. That’s made a huge difference in 


* Increase their ownership for improvement momentum. He’s still in charge, but he really has 


¢ Take on shared leadership roles. 


empowered different people in leadership roles 
and gotten many people involved. That has made 
* Build school capacity to sustain the changes a huge difference in that school in terms of 
cohesiveness and community.” 

- MLD Leadership Coach 


goals and action plans. 


over time, and sometimes despite leadership 
change. 


¢ Provide internal professional development as a content expert 
Key ingredients of collaborative leadership: 
¢ Cultivate a collaborative culture between the administration and teachers. 


¢ Develop the right person for the right job by encouraging teachers to take a lead in areas that 
interest them and by giving them support to try something new. 


¢ Provide teacher leaders with decision-making power and autonomy. 


¢ Develop structures for teachers to work together (e.g., leadership team, interdisciplinary 
teams, PLCs). 


e Establish time for regular collaboration and reflection. 
¢ Determine a common goal and action plan. 


¢ Define roles for team members. 


KEY FEATURES OF LEADERSHIP TEAMS 

Leadership teams were already in place at several MLD Project schools before the project 
began, while at other school, they were established at the beginning of the project. All were 
organized as a result of implementing a collaborative leadership structure at each school. 
Although MLD Project schools used varying methods of structuring their leadership teams, 
organizing their activities, and establishing meeting frequency and duration, several key features 
of the teams were common across schools. 
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Best practices for leadership team composition: 


¢ Consider whether appointments or volunteers are best for your school. 


* Goal is that members represent the whole faculty. 


¢ Include a district representative. 


Best Practices for leadership team organizational protocol: 


¢ Establish group expectations and norms. 


¢ Regular meeting schedule with agendas shared before 
meetings. 


¢ Document decisions and next steps. 
Strong leadership teams: 

e Are engaged and focused on improvement. 
¢ Examine data to drive actions. 

¢ Have acontinuous improvement approach. 


¢ Communicate actively with the whole faculty. 


“The best way to improve our 
climate was through the 
leadership team because they 
disseminated information to their 


departments about what we 
were planning and the positive 
approaches to addressing 
challenges.” 


- MLD Principal 


¢ Build capacity for sustainability beyond the grant or principal turnover. 


e Are constantly growing as they learn from challenges and setbacks. 


TRANSFORMATIVE SUPPORT AND RESOURCES FOR PRINCIPALS 


Through their participation in the MLD Project, principals received support and resources 


that they considered to be transformative to their leadership practices, beliefs, and behaviors. 


Principals reported that it was the combination of resources and supports that allowed them to 


work on the global culture and environment of their school, build collaborative leadership, work 


with teachers on instructional practices and interventions, and implement needed programs and 


practices (e.g., advisory, PLCs, discipline policy). 


Key supports and resources for principals: 


* Knowledge of middle-level philosophy, young adolescent needs, and best middle-grades 


practices. 


¢ Networking opportunities with other principals. 
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¢ Self-assessment of leadership behaviors and skills and development of a professional growth 


plan 
“| think for me the biggest piece was just 
* Refining a leadership style built around to increase my knowledge about what 
collaborative practices. works in the middle grades and how we 


= : ; can best implement those practices in 
¢ Visits to other high-performing schools. my school setting.” 


- MLD Principal 


VALUE OF VISITS TO STW SCHOOLS 

STW visits allowed groups of teachers from MLD Project schools to share a common 
experience by observing how a STW school had translated the STW Rubric’s “best” practices into 
“everyday” practices. The visits were powerful as they allowed for a better understanding not only 
of the best practices described in the rubric’s criteria, but the difference between implementing 
with “Good Quality” and “High Quality.” The visits encouraged schools that they were already 
doing some best practices and once they began consistently implementing additional best 
practices and seeing positive results, their confidence increased. 


Reasons why visits to STW schools were so impactful: 
¢ Powerful to visit another school and see best practices in action. 
¢ Validates existing best practices in project school. 


* Gain knowledge about how another school is + geal thatthe vidie to STW ech ols 


have provided a way for positive 
change to occur as the faculty sees 
other schools with similar 


successful. 


¢ Observe the difference between “good quality” 
versus “high quality.” 


populations and how they have 
made changes to bring about 
growth.” 

* Take away the practices that could be adopted or - MLD Leadership Coach 


modified to meet the project school’s needs. 


¢ Faculty on visit shares a common experience. 


¢ Encourages faculty to take ownership of improvement. 
¢ Validates the practice of utilizing the STW Rubric as a vision and framework for the school 


e [Illustrates what the STW Rubric looks like in daily practice when implemented successfully 
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IMPORTANCE OF NETWORKING 

MLD Project schools valued the knowledge, resources, and support gained through 
networking opportunities and professional development provided at the state and national levels 
through the Forum’s STW Network. For teachers, this meant increased opportunities to connect 
and learn from other teachers. Principals emphasized the benefits afforded to them from getting 
together with other principals to collaborate and share experiences and challenges. Project 
schools attended state and national middle-level conferences, as well as the annual national STW 
Conference. Rural schools especially valued these learning opportunities and viewed them as a 
lifeline from their rural isolation. 


Value gained from networking with other schools: 
* Opportunity to collaborate and share ideas. 
¢ Creation of a professional development cohort between project schools. 


* Facilitates a train the trainer “The grant provided the leadership team with 


model for professional supports to improve the capacity to instill change 
development. within the school and it provided us with 
opportunities to receive feedback. Additionally, the 
* Provides inspiration to make networking it provided has been invaluable and 


meaningful changes. connected our school with other middle schools 
across the state. It has been the springboard to 
change for the benefit of our students and will 
adopted or modified for school. create a lasting impact on the school.” 

- MLD Principal 


¢ Take away practices that could be 


* Gain knowledge about how 
another school is successful. 


* Opportunities to present at state or national conference as part of the MLD Project. 


¢ Builds school’s expertise and confidence. 


SCHOOL DISTRICT SUPPORT 
The MLD Project built 
requirements into the structure of 


“The superintendent bought in early on this 
process. He was at many of our meetings, when we 
were meeting with PLCs and with the leadership 
the project's activities and team. He would make time to come and be part of 
components that were intentionally that, and he supported our principal in that school 
designed to include district from day one and that’s very critical. Everybody 
needs to get in on the game. 


representatives in the process. The 
- MLD Leadership Coach 


most visible requirement was that a 
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district person be a regularly attending member of each school’s leadership team, thus cultivating 
school district involvement and support of the project from the beginning and maintaining it 
throughout the project. Principals believed that through a better understanding of STW, the school 
district would recognize that the rubric captured all of the district’s goals and vision. 


School district support was impactful in meaningful ways: 


¢ Importance of the school district providing the principal with the autonomy to make decisions 
and implement change. 


¢ School district representation on the school leadership team. 
¢ Cultivate continued school district support by: 


o Frequent communication ora 
presentation on the school’s “In the second year, my coach presented the 
progress that we had made and what to expect 
from the mock STW visit to the superintendent. 
Then the superintendent came to the mock STW 


progress as a result of the grant to 
the school district; 


visit and walked around with the visiting team. | 
think that really helped in seeing this as a free 
opportunity to make us better.” 

- MLD Principal 


o Inviting superintendent 
participation in the school’s mock 
visit with access to the expertise of 


the visiting national team of 
middle-grades experts; and 


o Inviting superintendent attendance to an AMLE presentation on the project or the STW 
Conference. 


¢ Emphasize school improvement supports would be provided at “no cost” to the school or 
school district. 


BUILDING SCHOOL CULTURE 

The first area of work that was addressed by the MLD Project leadership coaches was 
working with principals and leadership teams to build a positive, supportive, reflective, and 
student-centered school culture. The contextual changes in culture and climate were viewed as 
pre-requisites for leadership change, collaboration opportunities, and instructional 
improvements. In other words, school culture was the building block to improvements in other 
areas of the school. 


Key ingredients when building school culture: 
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¢ Transitioning from teacher-centered instruction to student-centered learning through student 
engagement and cooperative learning strategies. 

* Creating consistent expectations and a better understanding of student behavior. 

¢ Providing an adult advocate for each student and creating personal connections with students. 

* Creating a supportive culture that “We’re moving from an adult-centered to a 

allows risk-taking for teachers and student-centered facility and programs. The piece 

students. that has really bought us growth from a teaching 

standpoint is active engagement strategies for 

youngsters in the classroom and that is really 

their own education. elevating the teaching and learning. That is critical. 

They absolutely engage kids. We’ve got a group of 

kids that have to be active, so might as well channel 


it in a useful way.” 
- MLD Leadership Coach 


¢ Encouraging student investment in 


¢ Developing a growth mindset among 
teachers and students. 


¢ Empowering teacher leaders with 
decision-making power and autonomy. 


¢ Scheduling time regularly for reflective conversations on teaching and building-level practices. 
¢ Building a trusting environment to improve teaching practices through peer observations. 


¢ Being intentional with school data to secure teacher buy-in for implementation. 


BENEFITS OF SCHOOLS PARTICIPATING IN MOCK STW VISITS 

MLD Project school principals, leadership teams, and teachers unequivocally indicated that 
their participation in mock STW visits at their school was the activity that taught them the most 
about their improvement progress and goals. Mock STW visits involved the MLD Project schools 
hosting a national team of STW-trained visitors whose purpose was to spend a day touring the 
school, meeting with administrators and teachers, observing classrooms, and interviewing 
students and parents. The trained teamed used the STW criteria as the framework for their 
observations and they provided the school with a comprehensive report of their strengths and 
challenges at the conclusion of the visit. Preparing for such a visit, participating in the visit, and 
receiving the feedback from the national team was positive and motivating experience. 


Important elements of the mock STW visits to schools: 


¢ Improved teaching practices as schools monitored teaching practices more critically utilizing 
the STW Rubric in preparation for the mock STW visit. 


Page 57 MLD Project Final Evaluation Report CPRD, 2018 


LESSONS LEARNED 


Received expertise from a national team of middle-grades professionals. 


Validated existing best teaching practices. 
Confirmed principal and leadership team assessments and decisions. 


Resulted in action plans for future improvements by reviewing feedback and addressing 
mediating questions to identify and prioritize growth areas. 


Helped schools to align their school improvement action plans with the STW Rubric. 


Utilized the mock STW visit process and feedback as a “pre-visit” to assess readiness for 
applying for a STW designation. 


Built school confidence that they could become a nationally recognized STW School. 


BENEFITS OF PROJECT TEAM DOING MOCK VISITS FOR THEIR UNDERSTANDING OF SCHOOLS 


The MLD Project team, including leaders from the National Forum, leaders from Kentucky and 


Michigan, and other project partners expressed the growth they experienced by visiting the MLD 


Project schools to conduct the mock STW visits. They described the experience as enlightening to 


see the schools firsthand and experience the challenges they face as well as the successful 


implementation of the MLD Project. This experienced and trained national team recommends this 


practice to other project teams because of its value to both the schools and the project leadership. 


Important elements of the mock STW visits to project team: 


Increased familiarity and ownership of all the schools participating in the MLD Project. 


Identification of similar challenges (and successes) across project schools for future project- 
level discussions and decisions. 


State directors developed a deeper understanding of their state’s school through a fresh 
perspective from experienced colleagues. 


Increased familiarity of each state’s expectations for their project schools and coaches. 
Built comradery between project team members through a shared experience. 


Resulted in a positive impact on the work of the project team. 


KEY COMPONENTS OF PRINCIPAL EFFECTIVENESS 


The results of the data analyses presented in this report are positive in that they highlight 


numerous improvements at MLD Project schools. The results also provide unique insight into a 
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middle-grades school improvement project focused on principal leaders and collaborative 
leadership. The figure below (Figure 7) depicts the key supports, activities, and practices 
implemented at MLD Project schools that were the most impactful on building middle-grades 
leadership effectiveness. They provide a road map for other middle-grades schools and principals 
that are struggling to improve. With key supports, activities, and practices at the school level, by 
the principal, through a collaborative leadership structure and in a positive teaching/learning 
culture, middle-grades leadership is more effective. 


Figure 7. MLD Project Key Components of Middle-Grades Principal Effectiveness 


School Environment: Principals: 
- Adoption of STW guiding vision for middle- - Knowledgeable about young adolescents 
grades high performance - Committed to developmentally appropriate 
- Use of STW criteria for continuous middle-grades practices and structures 
improvement planning and monitoring - Serve as an instructional leader 
= Intentional use of examining school data to - Connected to a network for resources, best 
improve teaching practices practice, and visits to other schools 


- Use of reflective practices 


Middle-Grades 


Leadership 


Collaborative Leadership: Teaching and Learning Culture: 
- Development of teacher leaders - Student-centered learning for all 
- Implementation of structures for collaboration - Consistent and high expectations 
- Establish time for regular collaboration - Build strong relationships with students 
- Build shared capacity - Engaging and rigorous instruction 


- Involve district in improvement efforts - Growth mindsets among teachers and 
students 


Although the MLD Project results do not yet connect the contextual improvements in 
culture and climate, collaborative leadership practices, and the implementation of “best” middle- 
grades practices with student achievement growth, it is encouraging to find these improvements 
in programs and practices because improvements in these areas are the building blocks to 
improvements in student achievement. Student achievement cannot be improved without first 
changing and improving the school environment and the teaching and learning process. It is also 
encouraging to find student achievement growth at several MLD Project schools as this may 
indicate a trajectory toward more future growth. Finally, the fact that ten of the twelve project 
schools were designated as STW during the project or no-cost extension is another indicator of 
future achievement growth. 
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Sustainability 


The project leadership team, state leaders, leaderships coaches, and principals involved in 
the i3 STW Project expressed a belief that project schools will not only sustain the improvements 
they have made, but continue to grow and improve, particularly in the area of student 
achievement. Sustainability by definition is the “capacity to endure” and although the evaluation 
cannot predict whether project schools will sustain their accomplishments and continue to grow, 
data collected on the project highlight several key structures and practices that are likely to 
support sustainability and continued growth. 


STW DESIGNATION 

As described previously and mentioned in the prior section, the framework for the MLD 
Project was the Forum’s STW criteria. At a national level, the Forum uses the STW criteria for 
identifying middle-grades schools across the country that are on an upward trajectory of growth 
and continuous improvement. Schools that are interested in being designated as a STW by the 
Forum engage in an application process with the Forum’s affiliate in their state. The application 
process is comprised of several steps, including: completing an online application form; 
submitting data on student practices and performance; completing the online STW rubric; and 
hosting a visit to their school by a STW review team that includes interviews with the 
administrators, teachers, students, and parents; observations of classrooms; and team meetings. 
Typically, each of the 18 STW state affiliates designates 3 to 5 schools per year as STW schools. 


Although being designated as a STW was not articulated as an outcome for MLD Project 
schools by the end of the grant, it was understood that the project would lay the foundation for 
schools to be designated in the future. It turned out, however, that seven project schools made 
substantial improvements in practices and outcomes during the grant period and so applied and 
were designated as a STW school before the end of the grant. Three more project schools applied 
during the no-cost extension of the grant and were also designated as STW. They join more than 
300 other schools across the country that are STW schools and they serve as success stories for 
the MLD Project. Because the STW process requires all schools to apply for re-designation every 
three years, sustainability is supported through a continuous improvement mindset. STW schools 
use the STW criteria and rubric to regularly examine and reflect on ways to continue to improve. 
As stated on the STW rubric “even when the ultimate goal is reached, a true high performing 
middle school will continue to seek ways to improve as new challenges arise.” Sustaining the 
improvements and continuing to grow, therefore, is very likely among the MLD Project schools. 
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The ten (of the twelve) project schools that were designated as Schools to Watch Schools 


by the National Forum to Accelerate Middle-Grades Reform are: 


2017 

¢ Garrard Middle School, KY 

¢ Marshall Greene Middle School, MI 
¢ Mayfield Middle School, KY 

¢ Oscoda Middle School, MI 

¢ Richfield Public School Academy, MI 
* Rowan County Middle School, KY 

¢ Union County Middle School, KY 
2018: 

¢ Grant Middle School, MI 

¢ Mt. Morris Middle School, MI 


¢ Reese Middle School, MI 


SUSTAINABILITY PROTOCOL AND MATRIX 


“| know whoever joins that team or 
leaves that team, that somebody’s 
going to be on that team that says 
this is what we’re doing that’s great; 


it’s working and we’d like to 
continue it. And I’m sure they’re 
going to continue the work.” 

- MLD Principal 


In thinking about how schools could sustain best practices and momentum beyond the 


grant, the MLD Project developed a sustainability protocol and matrix. The protocol included a 


three-step process: 1) discuss how to plan for sustainability; 2) complete the sustainability matrix; 


and 3) develop a plan for sustainability. 


The first step was to begin a discussion about sustainability at the school utilizing an 


existing group (i.e., leadership team, interdisciplinary team, grade-level team, PLCs) or the entire 


faculty. This group would focus on why sustainability was important and what challenges might 


prevent sustainability in their school in order to strategize on what resources might be needed 


and how the school would evaluate success. 
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The second step was to complete the “Creating a sustainability matrix with the 


sustainability matrix. First, the group would four domains of the STW Rubric to complete 
identify all new practices they had implemented a plan for 2018 school year went very well. 
at their school as a result of their participation in All faculty had a voice in determining next 
the MLD Project. Then the group would identify steps using the sustainability matrix as a 


tool. We were able to get a sense of what 
the faculty valued, what was not necessary 
to continue, and the next steps in what they 
need to continue to work on to advance 
other voluntary initiatives to generate a their school forward.” 


comprehensive list. Finally, the group would - MLD Leadership Coach 


any other practices and programs that were 
currently being implemented at their school, 
such as district initiatives, state mandates, or 


discuss each practice (or program) on the list 
individually until they reached a consensus on the degree of implementation at the school and the 
level of impact on the child. 


To facilitate this conversation, the group utilized Figure 8. MLD Project Sustainability Matrix 
the sustainability matrix as a graphic organizer with 
four quadrants based on the practice’s impact on the Sustainability Matrix 
child (high/low) and the degree of implementation at 
their school (high/low) to determine where to assign 
each practice (Figure 8). Each quadrant of the 
sustainability matrix was assigned a priority category of 
low, medium, or high with an accompanying 
recommendation. A high priority practice had a high 


impact on the child and was also high in the degree of 
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implementation, and therefore the recommendation 
would be to sustain this practice. A practice that hada 


high impact on the child, but a low degree of Low Degree of Implementation High 
implementation, received a medium priority and the 


° 
= 


recommendation to invest in the fidelity of implementation to improve the practice. A practice 
with a high level of implementation, but a low impact on the child, was also considered a medium 
priority, but the recommendation was to investigate further and gather data to determine if the 
practice should be sustained or discontinued. Finally, a practice that had both a low impact on the 
child and a low degree of implantation was considered a low priority with the recommendation to 
discontinue the practice. The intention of step two was to provide everyone in the group witha 
voice for a shared, reflective conversation about school practices, and the expectation was that 
some practices would be reassigned to a different quadrant after these discussions. 
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The third and final step was to develop a plan for sustainability by aligning the newly 
prioritized practices to the school’s improvement plan. In this step the group revisited available 
resources and potential challenges to determine which practices they could sustain and then they 
incorporated those practices into their school improvement plan. Since continuous school 
improvement is the objective, targeted goals were to be set and monitored through data collection 
with plans to revisit the sustainability protocol with future school improvement planning. 


The MLD Project leadership team provided training on how to conduct the sustainability 
protocol to project schools and leadership coaches. Then the leadership coaches worked with 
their individual schools to facilitate the protocol and develop a sustainability plan, which was 
submitted to the MLD Project leadership team before the end of the grant. 
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Appendices 


APPENDIX A. MLD PROJECT FIDELITY OF IMPLEMENTATION MATRIX 


Component 1: Create a powerful vision for high performance using the STW criteria 


STW Criteria training for 


establishes a vision 


Faculty supports the 


Orientation/meeting each year for school 

representatives and coaches (mentors optional) 0 (low)=not attended 

to: 1) Provide a project overview for the year; 2) 1 (med)=1 school representative 
Go over the STW criteria; 3) Introduce the attended 

support team; 4) Go over the responsibilities of 2 (high}=more than 1 school 


STW Criteria training i year for school 

representatives and district leaders from each 0 (low)=not attended 

school to provide an in-depth understanding of 1 (med)=1 school representative 
the STW Criteria and to continue to learn tools attended 

for using the criteria. Training may occur in a 2 (high)=1 school representative and 
group setting or be conducted by coach at the district representative attended 
school. 

The MLD leadership coach works with the school 

leadership team to revise or establish their vision 

for high performance based on the STW criteria 0 (no)=not completed 

and a model of continuous improvement by the 4 (yes)=completed 

end of Year 1. In Years 2-3, the vision is 

revisited and revised as needed. Vision is written 

out and included in i 

The degree to which the faculty supports the 

school vision via their response to a Self-Study 0 (low)=<50% of faculty support vision 
Survey question (I support the vision/goals/ 1 (med)=50-75% of faculty support 
practices of the project). isi 


Implementation of component with fidelity= 
At least 70% of schools: Year 2 with med or high implementation (score of 1 or 2): 
Year 3 with high implementation (score of 2) 
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Component 2: Engage in assessment and planning to identify needs, develop goals, and implement an action plan 


The STW Rubric is a tool for continuing 
Faculty complete the improvement. It is divided into 4 sections 0 (low)=<50% faculty completed 
STW Rubric depicting the STW criteria. Each section contains | 1 (med)=50-75% faculty completed 
detailed examples of excellence. Teachers rate 2 (high)=76%+ faculty completed 
their implementation. 
The MLD leadership coach works with the 
Identify needs with leadership team to examine data (STW Rubric 
school leadership team | results, Self-Study Staff Survey results, student 0 (no}=not completed 
outcome data, etc.) and reflect on the STW 1 (yes)=completed 
criteria in order to identify areas for 
improvement. 
Each year, the MLD leadership coach works with 
Develop growth targets | the leadership team to develop growth targets for 0 (no)=not completed 
and action plan with each area of improvement, with an associated 1 (yes)=completed 
school leadership team | action plan for each. Schools follow state 
requirements for documentation to avoid 
duplicating efforts. 


Faculty input and The leadership team presents the action plan to 0 (no)=not completed 
support of action plan —_‘| the faculty for input and support. 1 (yes)=completed 


The MLD leadership coach works with the 
Implement the action leadership team and faculty to implement the 
plan action plan. Performance benchmarks are 0 (low)=<50% of action items 
included in the action plan in order to assess completed 
progress, monitor results, and make mid-course 2 (med)=50-75% of action items 
corrections. Approximately 2-4 goals per year. completed 
4 (high)=76%+ of action items 


Range of 0-9 
0 (low)=0-1 
1 (med)=2-7 


Implementation of component with fidelity= 
At least 70% of schools with high implementation (score of 2) 
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Component 3: MLD leadership coach works with leadership team 


Orientation/training for 
MLD leadership 
coaches 


MLD leadership 
coaches work with 
leadership team 


MLD leadership 
coaches work with 
principal to develop a 
professional growth 


MLD leadership 
coaches participate in 
ongoing training 


Orientation/training meeting for MLD leadership 
coaches at the beginning of each year (or when 
anew coach is hired). Meeting agenda includes 
project overview, STW criteria overview, 
coaching activities and expectations, and 
calendar of project events. 


Each year, MLD leadership coaches meet with 
the leadership team a minimum of once per 
month in person at the school from Sept to May. 


Each year, MLD leadership coaches work with 
the principal to develop a professional growth 
plan for improving leadership skills based on Val 
Ed leadership skills assessment data. 


Ongoing training includes participation in coach 
conference calls, networking via Edmodo or 
other mechanisms, state meetings, attendance 
at the national STW Conference, and going on 
STW visits. 


Implementation of component with fidelity= 
At least 70% of schools with high implementation (score of 2) 


0 (no)=not completed 
1 (yes)=completed 


0 (low)=0-4 meetings per year 
2 (med)=5-7 meetings per year 
4 (high)=8 or more meetings per year 


0 (no)=not completed 
4 (yes)=completed 


0 (low)=0-3 events per year 
1 (med)=4-5 events per year 
2 (high)=6 or more events per year 


Range of 0-11 
0 (low)=0-2 


Component 4: Principal mentor works with principal 


Each state provides mentoring experiences for 
project principals. This can take the form of 
principal mentors who work with project 


Principal mentors work 
with principals 


principals by meeting with them (in person or via 


phone or email). It can also take the form of 

mentoring that occurs for project principals as 
part of visits to other schools, visits to STW, or 
going o on STW designation visits. Menionng is 


Implementation of component with fidelity= 
At least 70% of schools with high implementation (score of 2) 


0 (low)=0-1 meetings per year 
2 (med)=2-3 meetings per year 
4 (high)=4 or more meetings per year 
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Component 5: Network school works with school 


Network schools host visits from the principal 

1 | School visits network and leadership team members, providing 

school observation opportunities (e.g., grade level 

meetings, team meetings, classrooms). 
discussion opportunities, and share resources. 
This manifests with either visits to STW sites, 
visits to or attendance on STW designation 
visits, visits to other i3 schools, or visits to other 
high performing schools. Schools go on a 
minimum of 2 visits per year. 


Implementation of component with fidelity= 
At least 70% of schools with med or high implementation (score of 1 or 2) 


0 (low)=no visits 
2 (med)=1 visit 
4 (high)=2 or more visits 


Component 6: School engages in a leadership professional learning community 


Each school establishes a leadership team 
School establishes a composed of a minimum of 6 people including 
leadership team the principal, district administrator, 3 teacher 
leaders, and one other staff member (e.g., 
counselor, assistant principal, curriculum 
coordinator). 
Training at the beginning of each year for 
Training for leadership | leadership team members. Training agenda 
team on functioning as_ | includes definition of a PLC (collegial group of 
a professional learning | administrators and teachers who work together 
community (PLC) regularly to collaborate and provide leadership 
for the school), operational procedures, group 
norms, and communication mechanisms. 
Leadership team members attend meetings 
Regular attendance of | regularly (twice per month in Oct, Jan, Feb, Mar, 
leadership team Apr, and May; once per month in Sep, Nov, and 
members at leadership | Dec.), with few absences. 
team meetings 


Implementation of component with fidelity= 
At least 80% of schools with high implementation (score of 2) 


0 (low)=not established 
1 (med)=established, but not with 6 
members per definition 
2 (high)=established with all 6 members 
per definition 


0 (no)=not completed 
1 (yes}=completed 


0 (low)=Average attendance at 
meetings is less than 50% of members 
2 (med)=Average attendance at 
meetings is 50-75% of members 
4 (high)=Average attendance at 
meetings is 76-100% of members 
Range of 0-7 
0 (low)=0-2 
1 (med)=3-5 
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School attends national 
STW Conference 


School attends i3 MLD 
meeting at national 
STW Conference 


School attends the state 
middle school 
conference or regional 
middle school workshop 


Operational Definiti 
for 
Indicator 


At least one person from the school attends the 
annual national STW Conference, attending 
sessions based on the areas for improvement 
identified as part of developing the action plan. 
Information is brought back to the school and 
reported to the full leadership team. 

At least one person from the school attends the 
i3 MLD project meeting at the annual national 
STW Conference. The meeting, hosted by the 
state project directors, is designed to develop 
collaborative relationships between schools and 
promote discussion of project challenges and 
SUCCESSES. 

At least one person from the school attends the 
state middle school conference (Michigan) or 
regional middle school workshop (Kentucky) 


Implementation of component with fidelity= 
At least 70% of schools with high implementation (score of 2) 


0 (low)=not attended 
1 (mea)=attended, but information not 
Shared with leadership team 
2 (high)=attended and information 
shared with leadership team 


Year 2: 
0 (no)=not attended 
1 (yes)=attended 


Year 3: 
NA 


0 (no)=not attended 
1 (yes)=attended 


Year 2: 
Range of 0-4 
0 (low)=0 
1 (med)=1-2 
2 (high)=3-4 


Year 3: 
Range of 0-3 
0 (low)=0 
1 (med)=1-2 

2 (high)= 


SS ee 
Component for Implementation Levels 
Indicator 


Component 8: School receives regular and systematic professional development trainings based upon needs and 
data from the evaluation process 


School leadership team 
attends a professional 
development training on 
leadership, 
collaboration, and STW 


Schools participate in 
professional 
development focused 
on their identified needs 


Each year, state project teams or coaches 
conduct a professional development session 
focused on leadership, collaboration, and STW 
with a focus on continuous improvement. At least 
2 people from the school's leadership team 
attend the session. 

MLD leadership coaches and state teams 
provide/arrange for professional development 
focused on the schoofs identified needs, such as 
student engagement, differentiated instruction, 
technology, and creating a personalized learning 
environment. A minimum of 2 sessions per year 
with at least 3 to 5 faculty from the school 
attends. 


Implementation of component with fidelity= 
At least 70% of schools with high implementation (score of 2) 


0 (no)=not attended 
1 (yes)=attended 


0 (low)=0 sessions attended 
1 (med)=1 session attended 
2 (high)=2 or more sessions attended 


Page 69 


MLD Project Final Evaluation Report 


CPRD, 2018 


APPENDICES 


APPENDIX B. MLD PROJECT K-PREP AND M-STEP ACHIEVEMENT FOR OVERALL STATE AND 
PROJECT SCHOOLS BY YEAR 


Reading K-PREP Results : - 
—_____ 2017 % of Proficient or Advanced Growth 
2015 to 2017 


| 

3 
[eon | |] 
| 
a 
| 
[fe 
seis 
feats 
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[se 
fone 


50.9% | 47.9% 1.6% 
is = i Higher than the overall state 


Mathematics K-PREP 
Results 2015-2017 


school2 | (34.8% 

fschool3 | (28.7% 
[School4 
fSchoolS 


% of Proficient or Advanced Growth 


37.1% 38.1% 
51.6% 58.8% 


a 
Overall State 43.2% 50.2% 49.1% 
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7* Grade 


Overall State 40.9% 45.4% 43.3% 


School 1 38.0% 


fsted@ | 8 
ies 
séecte | 
séeas 
teas ee 


8" Grade 


Overall State 44.2% 45.5% 48.7% 


School 1 46.8% 


fod? 
tots | 87 
ste 
seca 
sees 


| | Higher than the overall state 


ELA M-STEP Results = 
% of Proficient or Advanced Growth 
2015-2017 


[school9 | 200 | 200 | 222 | 22% 
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Higher than the overall state 


Page 74 MLD Project Final Evaluation Report CPRD, 2018 


APPENDICES 


APPENDIX C. MLD PROJECT IMPACT STUDY MODEL FOR ELA/READING 


Computing standard errors: 


Mixed-effects REML regression Number of obs = 5,712 
Group variable: Sch_ID Number of groups = 58 
Obs per group: 
min = 31 
avg = 114.2 
max = 198 
Wald chi2(15) = 6303.57 
Log restricted-likelihood = -5668.0208 Prob > chi2 = 8.28080 
ReadSS _Z8 | Coef. Std. Err z P>|z| [95% Conf. Interval] 
wee ee ee ee oe ee ee ee ee ee ee ee ee ee ee $---- nn nn rr rr rr rr rr rr rr rr rrr eee 
Intervention | -@43147 = .@551131 @.78 0.434 - .0648727 - 1511667 
ReadSs _Z5 | -6137239 .@096427 63.65 0.000 - 5948245 - 6326233 
AA | -.2508199 .@408064 -6.15 0.000 -.3307989 -.1708408 
Hisp | -.0305184  .0384307 -@.79 0.427 -.1058413 - 0448045 
Other | -0342738 .0401967 @.85 0.394 - .0445102 - 1130578 
female | -2182965 .@172052 12.69 0.000 - 184575 - 252018 
MI | -.2103298  .0976617 -2.15 0.031 -.4017433 -.0189163 
LEPStatus | -.02778@1 .0745085 -@.37. 0.709 -.1738141 - 1182539 
Special Ed | -.3411173 .0302466 -11.28 0.000 -.4003995 -.2818351 
Economically Disadvantaged | -.2016305 .@187118 -10.78 0.000 -.2383049 -.1649561 
Sch_AMO_AYP_c | -@203617 .0063698 3.20 0.001 - 0078771 - 0328463 
Sch_Enroll_c | -0882935 .0001826 1.61 0.108 - .0800645 - 0006514 
Sch_FRL_c | -.082464 .0026868 -@.92 0.359 - .00773 - 0828021 
Sch_AA c | -0870514 .8032808 2.15 0.032 - 0006211 - 0134816 
Sch_Hisp_c | 0004671 .@042532 Q@.11 0.913 - .087869 - 0888032 
_cons | . 1224304 - 048695 2.51 0.012 - 02699 - 21787089 
Random-effects Parameters | Estimate Std. Err. [95% Conf. Interval] 
wee wm em em em em em em em em em em em em em em em ew ee ew ew ee ee = S sieiesticestietietieetieticesticetieatieatieatieatteedtiestiatiedtietiedticeteadtiadteatcatiatatiatamtadtietateatetdtatedtatdteatcedtatattetatateatatiel 
Sch_ID: Identity | 
var(_cons) | -0213914 .@056025 - 0128028 - 0357414 
eee ee ee ee ee ee ee ee ee ee ee = te -- ee ee ee nn nr nn rn rn rrr rr rr rr rrr eee 
var(Residual) | -4122673 - 007754 - 3973465 -4277484 
LR test vs. linear model: chibar2(@1) = 152.77 Prob >= chibar2 = 0.0000 
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APPENDIX D. MLD PROJECT IMPACT STUDY MODEL FOR MATH 


Computing standard errors: 


Mixed-effects REML regression Number of obs = 5,737 
Group variable: Sch_ID Number of groups = 58 
Obs per group: 
min = 31 
avg = 114.7 
max = 196 
Wald chi2(15) = 7278.98 
Log restricted-likelihood = -5190.5025 Prob > chi2 = 8.9000 
MathSS_Z8 | Coef. Std. Err. z P>|z| [95% Conf. Interval] 
ee ee ee ee ae ae ae a ee ae ee ee ee ee ee ee Se saleesiteeticetietietieeatietietinenticetivetientiatieatieetieetitieatieatieatistietieticetticatietietatiatieetietiatiatietieteatettatatiedtiatnadtiettaettatnetiatataatatatatiatedtatatatatatnadadtied 
Intervention | -.@@04777 .0816594 -@.01 8.995 -.1605271 -1595717 
Mathss | | -6792261 .@092622 73.33 0.000 - 6610726 - 6973797 
AA | -.0934549 .@372165 -2.51 0.012 -.1663978  -.0205119 
Hisp | -.@525098  .@348092 -1.51 0.131 -.1207345 -0157149 
Other | -0324889 .0366206 @.89 0.375 - .0392863 - 104264 
female | -1430595 .0156584 9.14 @.000 - 1123697 - 1737493 
MI | -.1913631 .1447356 -1.32 0.186 - .4750397 + 8923135 
LEPStatus | -100634 .0667853 1.51 0.132 - .0302629 - 2315308 
Special Ed | -.1559898  .0272821 -5.7/2 0.000 -.2094618 -.1025179 
Economically Disadvantaged | -.1509614  .0@170786 -8.84 0.000 -.1844348 -.1174881 
Sch_AMO_AYP_c | -@235668  .0094069 2.51 0.012 - 0051296 - 042004 
Sch_Enroll_c | -080293 .0@02702 1.08 0.278 - .0802367 - 0808226 
Sch_FRL_c | -.0@27847 .0039545 -@.78 0.481 - .0105354 - 084966 
Sch_AA c | - .082999 - 084808 -@.62 0.533 - .0124224 - 0064245 
Sch_Hisp_c | -.0@18937 .0062427 -@.30 0.762 - .0141292 - 0103418 
_cons | -1042299 .0708356 1.47 0.141 - .0346053 - 2430651 
Random-effects Parameters | Estimate Std. Err. [95% Conf. Interval] 
we ww em em em em em wm em em em em em em em em em em em em em ew ee ee = Se eetieetieetietietieeticetieetietietietietiesticadtiattesticedtieatietietteatietietietiedttetitetadtietetadietetetedietdeditedttadtadtied 
Kch_ID: Identity | 
var(_cons) | -9526652 .0122423 - 0333932 - 0830595 
eee ee oe ee ee ee ee ee ee ee ee ee ee ee = $--- ee ee ee nr rr rr rr rrr rr rr rrr re eee ee 
var(Residual) | -3434574 .0064453 - 3310543 - 3563252 
LR test vs. linear model: chibar2(@1) = 543.00 Prob >= chibar2 = 0.0000 
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